File manager - Edit - /opt/gsutil/gslib/__pycache__/metrics.cpython-39.pyc
Back
a +(Wg� � @ s d Z ddlmZ ddlmZ ddlmZ ddlmZ ddlZddlmZ ddl m Z ddlZddlZddl Z ddlZddlZddlZddlZddlZddlZddlZddlZddlZddlZdd lmZ dd lmZ ddlZddlmZ ddlmZ dd l m!Z! ddl"m#Z# ddl"m$Z$ dZ%dZ&dZ'dZ(dZ)dZ*dZ+e�,d�Z-ej.�/ej.�0dd��Z1dZ2dddddd d!d"d#d$d%d&d'd(d)d*d+d,d-d.d/d0d1d2d3d4d5d6d7�Z3G d8d9� d9e4�Z5d:d;� Z6d<d=� Z7ej8e6d>d?� ��Z9e6dLd@dA��Z:e6dBdC� �Z;e6dDdE� �Z<e6dFdG� �Z=e6dHdI� �Z>dMdJdK�Z?dS )Nz:Static data and helper functions for collecting user data.� )�absolute_import)�print_function)�division)�unicode_literalsN)�defaultdict��wraps)�input)�urllib)�VERSION)�Metric)�system_util)�CalculateThroughput)�HumanReadableToBytesz(https://ssl.google-analytics.com/collectzUA-36037335-16zUA-36037335-17ZCommandZRetryableErrorZ FatalErrorZPerformanceSummaryz.*google\.com$�~z.gsutil/analytics-uuidZDISABLED�ec�ea�el�evZcd1Zcd2Zcd3Zcd4Zcd5Zcd6Zcd7Zcd8Zcd9Zcd10Zcm1Zcm2Zcm3Zcm4Zcm5Zcm6Zcm7Zcm8Zcm9Zcm10Zcm11Zcm12Zcm13Zcm14)zEvent CategoryzEvent ActionzEvent LabelzEvent Value�Command Name�Global Options�Command-Level Options�Config� Command Alias�Fatal Error�Parallelism Strategy�Source URL Type�Provider Types� Timestamp�Execution Time�Retryable Errors�Is Google Corp User� Num Processes�Num Threads�#Number of Files/Objects Transferred�!Size of Files/Objects Transferred�Average Overall Throughput�Num Retryable Service Errors�Num Retryable Network Errors�Thread Idle Time Percent�Slowest Thread Throughput�Fastest Thread Throughput� Disk I/O Timec @ s� e Zd ZdZeefdd�ZdZdZdd� Z e efdd��Ze d d � �Ze dd� �Ze d)dd��Ze d*dd��Ze dd� �Zdd� Zdd� Zeddfdd�ZG dd� de�Zdd� Zd d!� Zd"d#� Zd$d%� Zd+d'd(�ZdS ),�MetricsCollectorz�A singleton class to handle metrics reporting to Google Analytics (GA). This class is not thread or process-safe, and logging directly to the MetricsCollector instance can only be done by a single thread. c C s� t � | _|| _t�� | _t�� }tj � dd�}|r6t}| �� }d}t �t�� �rTd}ddd|d|d d td |td |i| _d jt�� t�� d�| _g | _tt�| _d| _dS )a Initialize a new MetricsCollector. This should only be invoked through the GetCollector or StartTestCollector functions. Args: ga_tid: The Google Analytics tracking ID to use for metrics collection. endpoint: The URL to send requests to. �GSUtilZuse_test_GA_propertyr � �v�1�tid�cid�t�eventr r! z{system}/{release})�system�releaseN)�_GetTimeInMillis� start_time�endpoint�logging� getLogger�loggerr- �_GetCID�boto�config�getbool�_GA_TID_TESTING�_ValidateAndGetConfigValues�_GOOGLE_CORP_HOST_RE�match�socket�gethostname� _GA_LABEL_MAP� ga_params�format�platformr6 r7 � user_agent�_metricsr �int�retryable_errors�perf_sum_params)�self�ga_tidr: r3 Zuse_test_property� config_valuesZis_corp_user� rT �/opt/gsutil/gslib/metrics.py�__init__x s0 � � zMetricsCollector.__init__Nc sL g � d�� �fdd�}dD ]\}}|||dd� d� qd�d D ]\}}|||�fd d�d� q>dD ]\}}|||dd� d� qbd D ]V\}}t j�||�}|r�zt|�}� �||f� W q� ty� � �|�f� Y q�0 q�|dddd� d� |dddd� d� |dddd� d� |dddd� d� |dddd� d� d�tdd� � D ���S )z�Parses the user's config file to aggregate non-PII config values. Returns: A comma-delimited string of config values explicitly set by the user in key:value pairs, sorted alphabetically by key. �INVALIDc s` z@t j�| |�}|r,||�r,� �||f� n|r>� �|�f� W n � �|�f� Y n0 d S �N)r? r@ � get_value�append)�section�category� validation_fn�config_value)rS �invalid_value_stringrT rU �GetAndValidateConfigValue� s zOMetricsCollector._ValidateAndGetConfigValues.<locals>.GetAndValidateConfigValue))�Boto�https_validate_certificates)r. �disable_analytics_prompt)r. Z use_magicfile)r. Ztab_completion_time_logsc S s t | ��� dv S )N)�true�false)�str�lower��valrT rT rU �<lambda>� s �z>MetricsCollector._ValidateAndGetConfigValues.<locals>.<lambda>)r[ r\ r] i� ))ra �debug)ra Zhttp_socket_timeout)ra Znum_retries)ra Zmax_retry_delay)r. Zdefault_api_version)r. Z%sliced_object_download_max_components)r. Zparallel_process_count)r. Zparallel_thread_count)r. Zsoftware_update_check_period)r. Ztab_completion_timeout)�OAuth2Zoauth2_refresh_retriesc s t | ��� ot| �� k S rX )rf �isdigitrN rh )�small_int_thresholdrT rU rj � s ))r. Zresumable_threshold)r. Zrsync_buffer_lines)r. Ztask_estimation_thresholdc S s t | ��� S rX )rf rm rh rT rT rU rj � � ))r. Z(parallel_composite_upload_component_size)r. Z#parallel_composite_upload_threshold)r. Z%sliced_object_download_component_size)r. Z sliced_object_download_thresholdr. Zcheck_hashesc S s | dv S )N)Zif_fast_else_failZif_fast_else_skip�always�neverrT rh rT rT rU rj s �Zcontent_languagec S s | � � ot| �dkS )N� )�isalpha�lenrh rT rT rU rj ro Zjson_api_versionc S s | d � � dko| dd � �� S )Nr r0 r/ )rg rm rh rT rT rU rj ro Z prefer_apic S s | dv S )N)�json�xmlrT rh rT rT rU rj ro rl Ztoken_cachec S s | dv S )N)Zfile_systemZ in_memoryrT rh rT rT rU rj s ��,c S s g | ]}d � |d |d ��qS )z{0}:{1}r r/ )rJ )�.0r@ rT rT rU � <listcomp> s z@MetricsCollector._ValidateAndGetConfigValues.<locals>.<listcomp>)r? r@ rY r rZ � ValueError�join�sorted) rQ r` r[ Z bool_categoryZsmall_int_categoryZlarge_int_categoryZdata_size_categoryr^ Z size_in_bytesrT )rS r_ rn rU rC � sn � ���������z,MetricsCollector._ValidateAndGetConfigValuesc C s"