hots.main

Entry point of hots module through hots path [OPTIONS].

• path is the folder where we find the files

  x container_usage.csv : describes container resource consumption x node_meta.csv : describes nodes capacities (x node_usage.csv : describes nodes resource consumption)

• type ‘hots –help’ for options description

The entire methodology is called from here (initialization, clustering, allocation, evaluation, access to optimization model…).

hots.main.add_time(loop_nb, action, time)

Add an action time in times dataframe.

Parameters:

• loop_nb (int) – current loop number

• action (str) – action / method we add time for

• time (float) – running time of the action

hots.main.analysis_period(my_instance, config, method)

Perform all needed process during analysis period (T_init).

Parameters:

• my_instance (Instance) – Instance object

• config (Dict) – parameters set

• method (str) – method used to solve initial problem

Returns:

tuple with Instance object, nodes data with new placement, clustering data and

clustering labels :rtype: Tuple[Instance, pd.DataFrame, pd.DataFrame, List]

hots.main.build_matrices(my_instance, tmin, tmax, labels_)

Build period dataframe and matrices to be used.

Parameters:

• my_instance (Instance) – Instance object

• tmin (int) – starting time of current window

• tmax (int) – final time of current window

• labels (List) – clustering labels

Returns:

current loop data, clustering data, dissimilarity matrix and adjacency matrices

Return type:

Tuple[pd.DataFrame, pd.DataFrame, np.array, np.array, np.array]

hots.main.close_files()

Write the final files and close all open files.

hots.main.end_loop(working_df_indiv, tmin, nb_clust_changes, nb_place_changes, nb_overload, total_loop_time, loop_nb, df_host_evo)

Perform all stuffs after last loop.

Parameters:

• working_df_indiv (pd.DataFrame) – current loop data

• tmin (int) – starting time of current window

• nb_clust_changes (int) – number of changes in clustering

• nb_place_changes (int) – number of changes in placement

• nb_overload (int) – number of node overload

• total_loop_time (float) – total running time for all loops

• loop_nb (int) – final loop number

• df_host_evo (pd.DataFrame) – evolving node data

Returns:

evolving node data

Return type:

pd.DataFrame

hots.main.eval_clustering(my_instance, w, u, clust_model, clustering_dual_values, constraints_dual, tol_clust, tol_move_clust, tol_open_clust, df_clust, cluster_profiles, labels_, loop_nb, solver)

Evaluate current clustering solution and update it if needed.

Parameters:

• my_instance (Instance) – Instance object

• w (np.array) – dissimilarity matrix

• u (np.array) – clustering adjacency matrix

• clust_model (mdl.Model) – clustering optimization model

• clustering_dual_values (Dict) – previous loop dual values for clustering

• constraints_dual (List) – constraints type compared to trigger conflicts

• tol_clust (float) – threshold used for clustering conflicts

• tol_move_clust (float) – threshold used for number of clustering moves

• tol_open_clust (float) – threshold used for opening new cluster

• df_clust (pd.DataFrame) – clustering related data

• cluster_profiles (np.array) – computed clusters mean profiles

• labels (List) – cluster labels

• loop_nb (int) – current loop number

• solver (str) – solver used for pyomo

Returns:

clustering variance matrix, number of changes in clustering, silhouette score before

and after the loop, clustering dual values, clustering optimization model, number of nodes and edges in conflict graph + max and mean degree (clustering) :rtype: Tuple[

np.array, int, float, float, Dict, mdl.Model, int, int, float, float

]

hots.main.eval_placement(my_instance, working_df_indiv, w, u, v, dv, placement_dual_values, constraints_dual, place_model, tol_place, tol_move_place, nb_clust_changes_loop, loop_nb, solver)

Evaluate current clustering solution and update it if needed.

Parameters:

• my_instance (Instance) – Instance object

• working_df_indiv (pd.DataFrame) – current loop data

• w (np.array) – dissimilarity matrix

• u (np.array) – clustering adjacency matrix

• v (np.array) – placement adjacency matrix

• dv (np.array) – clustering variance matrix

• placement_dual_values (Dict) – previous loop dual values for placement

• constraints_dual (List) – constraints type compared to trigger conflicts

• place_model (mdl.Model) – placement optimization model

• tol_place (float) – threshold used for placement conflicts

• tol_move_place (float) – threshold used for number of placement moves

• nb_clust_changes_loop (int) – number of changes in clustering in current loop

• loop_nb (int) – current loop number

• solver (str) – solver used for pyomo

Returns:

number of placement changes, dual values related to current placement, placement

optimization model, number of nodes and edges in conflict graph + max and mean degree :rtype: Tuple[int, Dict, mdl.Model, int, int, float, float]

hots.main.eval_sols(my_instance, working_df_indiv, cluster_method, w, u, v, clust_model, place_model, constraints_dual, clustering_dual_values, placement_dual_values, tol_clust, tol_move_clust, tol_open_clust, tol_place, tol_move_place, df_clust, cluster_profiles, labels_, loop_nb, solver)

Evaluate clustering and placement solutions.

Parameters:

• my_instance (Instance) – Instance object

• working_df_indiv (pd.DataFrame) – current loop data

• cluster_method (str) – method used to update clustering

• w (np.array) – dissimilarity matrix

• u (np.array) – clustering adjacency matrix

• v (np.array) – placement adjacency matrix

• clust_model (mdl.Model) – clustering optimization model

• place_model (mdl.Model) – placement optimization model

• constraints_dual (List) – constraints type compared to trigger conflicts

• clustering_dual_values (Dict) – previous loop dual values for clustering

• placement_dual_values (Dict) – previous loop dual values for placement

• tol_clust (float) – threshold used for clustering conflicts

• tol_move_clust (float) – threshold used for number of clustering moves

• tol_open_clust (float) – threshold used for opening new cluster

• tol_place (float) – threshold used for placement conflicts

• tol_move_place (float) – threshold used for number of placement moves

• df_clust (pd.DataFrame) – clustering related data

• cluster_profiles (np.array) – computed clusters mean profiles

• labels (List) – cluster labels

• loop_nb (int) – current loop number

• solver (str) – solver used for pyomo

Returns:

number of changes in clustering and placement, silhouette score before and after loop,

number of nodes and edges in conflict graph + max and mean degree (clustering and placement), optimization models and associated dual values, clustering related data, clusters mean profiles and cluster labels :rtype: Tuple[

int, int, float, float, int, int, float, float, int, int, float, float, mdl.Model, mdl.Model, Dict, Dict, pd.DataFrame, np.array, List

]

hots.main.loop_kmeans(my_instance, df_clust, labels_)

Update clustering via kmeans from scratch.

Parameters:

• my_instance (Instance) – Instance object

• df_clust (pd.DataFrame) – clustering related data

• labels (List) – clustering labels

Returns:

clustering variance matrix, number of changes in clustering, silhouette score before

and after loop, null objects to match other return objects :rtype: Tuple[

np.array, int, float, float, Dict, mdl.Model, int, int, float, float

]

hots.main.pre_loop(my_instance, working_df_indiv, df_clust, w, u, constraints_dual, v, cluster_method, solver)

Build optimization problems and solve them with T_init solutions.

Parameters:

• my_instance (Instance) – Instance object

• working_df_indiv (pd.DataFrame) – current loop data

• df_clust (pd.DataFrame) – clustering related data

• w (np.array) – dissimilarity matrix

• u (np.array) – clustering adjacency matrix

• constraints_dual (List) – constraints type compared to trigger conflicts

• v (np.array) – placement adjacency matrix

• cluster_method (str) – method used to update clustering

• solver (str) – solver used for pyomo

Returns:

optimization models and associated dual values

Return type:

Tuple[mdl.Model, mdl.Model, Dict, Dict]

hots.main.preprocess(path, k, tau, method, cluster_method, param, output, tolclust, tolplace)

Load configuration, data and initialize needed objects.

Parameters:

• path (str) – initial folder path

• k (int) – number of clusters to use for clustering

• tau (int) – time window size for loop

• method (str) – method to use to solve initial problem

• cluster_method (str) – method to use to update clustering

• param (str) – parameter file to use

• output (str) – output folder to use

• tolclust (float) – threshold to use for clustering conflict

• tolplace (float) – threshold to use for placement conflict

Returns:

tuple with needed parameters, path to output and Instance object

Return type:

Tuple[Dict, str, Instance]

hots.main.progress_time_noloop(instance, fixing, tmin, tmax, labels_, loop_nb, constraints_dual, clustering_dual_values, placement_dual_values, tol_clust, tol_move_clust, tol_place, tol_move_place)

We progress in time without performing the loop, checking node capacities.

Parameters:

• instance (Instance) – Instance object

• fixing (str) – method used to fix node assignment if overload

• tmin (int) – starting time of current window

• tmax (int) – final time of current window

• labels (List) – clustering labels

• loop_nb (int) – current loop number

• constraints_dual (List) – constraints type compared to trigger conflicts

• clustering_dual_values (Dict) – previous loop dual values for clustering

• placement_dual_values (Dict) – previous loop dual values for placement

• tol_clust (float) – threshold used for clustering conflicts

• tol_move_clust (float) – threshold used for number of clustering moves

• tol_place (float) – threshold used for placement conflicts

• tol_move_place (float) – threshold used for number of placement moves

Returns:

evolving node data + number of overloads + loop number + clustering and placement

changes :rtype: Tuple[pd.DataFrame, int, int, int, int]

hots.main.run_period(my_instance, df_host_evo, df_indiv_clust, labels_, config, output_path, method, cluster_method)

Perform all needed process during evaluation period.

Parameters:

• my_instance (Instance) – Instance object

• df_host_evo (pd.DataFrame) – evolving node data

• df_indiv_clust (pd.DataFrame) – clustering individuals data

• labels (List) – clustering labels

• config (Dict) – set parameters

• output_path (str) – output folder path

• method (str) – method to use to solve problem

• cluster_method (str) – method to use for clustering

Returns:

evolving node data and number of nodes overloads

Return type:

Tuple[pd.DataFrame, int]

hots.main.spec_params(config, list_params)

Define specific parameters.

Parameters:

• config (Dict) – initial parameters set

• list_params (List) – parameters list for threshold

Returns:

final parameters set

Return type:

Dict

hots.main.stream_km(my_instance, df_clust, labels_)

Update clustering via kmeans from scratch.

Parameters:

• my_instance (Instance) – Instance object

• df_clust (pd.DataFrame) – clustering related data

• labels (List) – clustering labels

Returns:

clustering variance matrix, number of changes in clustering, silhouette score before

and after loop, null objects to match other return objects :rtype: Tuple[

np.array, int, float, float, Dict, mdl.Model, int, int, float, float

]

hots.main.streaming_eval(my_instance, df_indiv_clust, labels_, mode, tick, constraints_dual, tol_clust, tol_move_clust, tol_open_clust, tol_place, tol_move_place, tol_step, cluster_method, solver, df_host_evo)

Define the streaming process for evaluation.

Parameters:

• my_instance (Instance) – Instance object

• df_indiv_clust (pd.DataFrame) – clustering individual data

• labels (List) – clustering labels

• mode (str) – mode used to trigger the loop

• tick (int) – number of new datapoints before new loop

• constraints_dual (List) – constraints type compared to trigger conflicts

• tol_clust (float) – threshold used for clustering conflicts

• tol_move_clust (float) – threshold used for number of clustering moves

• tol_open_clust (float) – threshold used for opening new cluster

• tol_place (float) – threshold used for placement conflicts

• tol_move_place (float) – threshold used for number of placement moves

• tol_step (float) – step to increment previous threshold

• cluster_method (str) – method used to update clustering

• solver (str) – solver used for pyomo

• df_host_evo (pd.DataFrame) – evolving node data

Returns:

figures to plot to see clustering and nodes evolution + evolving node data + number

of node overloads :rtype: Tuple[plt.Figure, plt.Figure, plt.Figure, List, pd.DataFrame, int]