| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| | |
|
| |
|
|
| |
tuiclient checks distfile status, and returns when distfile is downloaded or not in the queue.
|
| |
|
|
|
| |
Provide info on distfiles that are already in the list or just added,
but not in progress (not started yet or already downloaded) to ui_server/tui_clients.
|
| |
|
|
| |
Distfiles window shows progress on distfile downloads.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
[scripting_and_scheduling]
Segget provides Python scripting functionalyty to support scheduling.
Each time segget tries to start a new connection certain network it calls
a python script (client.py) to accept or reject this connection and
if necessary adjusts its settings.
PYTHON_PATH
Define path to python
Default:
python_path=/usr/bin/python
SCRIPTS_DIR
Define a path to the dir with python scripts. Before establishing connection for
a particular segment via network# segget checks SCRIPTS_DIR.
If SCRIPTS_DIR contains net#.py file, segget will launch schedule() function
from this file to apply settings for connetion and accept or reject this
segment for the moment. net#.py file is a python script file
with a user-writen schedule() function.
It's necessary to import functions before using get("variable"),
set("variable",value), accept_segment() and reject_segment() in schedule().
get() function can obtain values for the following variables:
connection.num, connection.url, connection.max_speed_limit,
network.num, network.mode, network.active_connections_count,
distfile.name, distfile.size, distfile.dld_segments_count,
distfile.segments_count, distfile.active_connections_count,
segment.num, segment.try_num, segment.size, segment.range
set() function can change connection.max_speed_limit, see example:
-----------------EXAMPLE STARTS-----------------
from functions import *
import time;
def schedule():
localtime = time.localtime(time.time());
hour=localtime[3];
# disable downloading distfiles that have size more than 5 000 000 bytes
# from 8-00 to 22-00.
if hour>8 and hour<22 and (get("distfile.size"))>5000000:
print "reject because distfile is too big"
reject_segment()
# set speed limit 50 000 cps for distfiles larger than 1 000 000 bytes
if get("distfile.size")>1000000:
print "limit connection speed"
set(connection.max_speed_limit, 50000)
accept_segment()
-----------------EXAMPLE ENDS-----------------
From example above localtime returns following tuple:
Index Attributes Values
0 tm_year e.i.: 2008
1 tm_mon 1 to 12
2 tm_mday 1 to 31
3 tm_hour 0 to 23
4 tm_min 0 to 59
5 tm_sec 0 to 61 (60 or 61 are leap-seconds)
6 tm_wday 0 to 6 (0 is Monday)
7 tm_yday 1 to 366 (Julian day)
8 tm_isdst -1, 0, 1, -1 means library determines DST
Therefore localtime[3] provides hours.
Segment will be accecpted by default if it was neither accepted nor rejected
during the schedule() function.
sagget saves logs of resulting stdout and stderr in the log folder
separatly for each network. Hence, if there's an error in net3.py file python
error message would be saved to net3_script_stderr.log. Results of print would
be saved in net3_script_stdout.log.
Default:
scripts_dir=./scripts
SCRIPT_SOCKET_PATH
Segget uses AF_UNIX domain sockets for communication with python.
Specify path for the socket on your filesystem.
Default:
script_socket_path=/tmp/segget_script_socket
|
| |
|
|
|
| |
Add option to network#.conf files:
[mode]
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
| |
0) Segget starts from the highest priority networks.
1) Segget tries local mirrors and mirrors provided by proxy-fetchers first.
2) If no success and requests to proxy-fetchers are allowed => segget selects proxy-fetcher and requests from it.
3) If proxy-fetcher replied that file has been downloaded => segget starts from step 1 again.
4) If no download started till now and remote-mirrors are allowed => segget selects a remote network and selects a mirror from it.
5) If no success on step 4, segget switches to lower priority networks
|
| | |
|
| |
|
|
|
|
|
|
|
| |
R_PF_BE_MORE_PATIENT
R_PF_ERROR_ADDING_TO_PROXY_QUEUE
R_PF_ADDED_TO_PROXY_QUEUE
R_PF_ALREADY_WAS_IN_QUEUE
R_PF_DOWNLOADED
R_PF_FAILED
|
| |
|
|
| |
don't add it.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
PROXY_FETCHER_IP
This option is active only when NETWORK_MODE=1, in other cases it's ignored.
Specify IP addres of the proxy-fetcher
Default:
proxy_fetcher_ip=none
PROXY_FETCHER_PORT
This option is active only when NETWORK_MODE=1, in other cases it's ignored.
Specify proxy-fetcher address
Default:
proxy_fetcher_port=3131
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
NOTE: Proxy-fetcher downloads distfiles requested by other segget daemons.
File will be ignored if it's already downloaded.
TO-DO: Proxy-fetcher needs to check if distfile is already in the download list.
[provide_proxy_fetcher_to_others]
PROVIDE_PROXY_FETCHER_IP
Define an ip address segget will use to provide access for tuiclients.
The parameter should be a string holding your host dotted IP address.
Default:
provide_proxy_fetcher_ip=127.0.0.1
PROVIDE_PROXY_FETCHER_PORT
Define a port segget will use to provide access for tuiclients.
The parameter should be an integer.
Minimum value: 1
Maximum value: 65535
Default:
provide_proxy_fetcher_port=9777
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
| |
to segget.conf file
SYNOPSIS: PROVIDE_MIRROR_FILES_RESTRICT_LIST_ON= 0 | 1
If PROVIDE_MIRROR_DIR=none this option will be ignored.
- If set to 1, segget will compare distfile name with the list of forbiden
patterns from the restricted.conf file. If distfile name contains any of the
patterns, no symlink will be provided to this distfile.
Default:
provide_mirror_files_restrict_list_on=0
|
| |
|
|
|
|
|
|
| |
[provide_mirror_to_others]
PROVIDE_MIRROR_DIR
Define a dir to store distfiles for mirroring.
Default:
provide_mirror_dir=./provide_mirror_dir
|
| |
|
|
| |
network_array[network_num].benchmarked_mirror_list
|
| |
|
|
|
|
|
|
|
|
|
|
| |
SYNOPSIS: ONLY_LOCAL_WHEN_POSSIBLE=0 | 1
If NETWORK_USES_OWN_MIRROR_LIST_ONLY_ON=0 this option will be ignored.
- If set to 1, segget will not use remote mirrors with equal or lower priority
until all mirrors in network0_mirrors.conf file have failed.
- If set to 0, segget will use remote mirrors with equal priority or mirrors
with lower priority when this network has NO free connections (see option
NETWORK_MAX_CONNECTIONS in [network_connections] section of this file).
Default:
only_local_when_possible=1
|
| |
|
|
|
|
|
|
|
|
|
| |
network#.conf :: [network_mirrors]
SYNOPSIS: NETWORK_USES_OWN_MIRROR_LIST_ONLY_ON=0 | 1
- If set to 1, segget will replace mirror list provided by portage system with
the list from network1_mirrors.conf
- If set to 0, segget will use ONLY mirror list provided by portage system,
and will NOT use the list from network1_mirrors.conf file
Default:
use_own_mirror_list_only_on=0
|
| |
|
|
| |
Network with the smallest active_connections_num wins.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Add [networks] section to segget.conf
[networks]
network0_priority=10
network1_priority=0
network2_priority=0
network3_priority=0
network4_priority=0
network5_priority=0
network6_priority=0
network7_priority=0
network8_priority=0
network9_priority=0
2) Following options moved from segget.conf to network#.conf
[network_bind]
bind_interface=none
[network_connections]
max_connections=2
connection_timeout=15
ftp_response_timeout=180
timeout=500
low_connection_speed_limit=1000
low_connection_speed_time=10
max_connection_speed=3000
[network_protocols]
[network_user_data]
user_agent=segget
[network_proxy]
proxy_ip_or_name=none
proxy_port=3128
proxy_user=none
proxy_password=none
proxy_off=1
[network_mirrors]
use_own_mirror_list_on=0
|
| |
|
|
|
|
|
|
| |
SYNOPSIS: del_pkg_list_when_dld_finished=0 | 1
- If del_pkg_list_when_dld_finished set to 1:
Segget deletes pkg.list file, after all distfiles were successfuly fetched.
Default:
del_pkg_list_when_dld_finished=1
|
| |
|
|
|
|
|
|
|
|
|
| |
CURRENT_SPEED_TIME_INTERVAL_MSECS
segget transfers may have bursty nature of their traffic. Therefore, while
measuring current speed, segget actually calculates average speed during
current_speed_time_interval_msecs time interval, defined in milliseconds.
Min limit:100
Max limit: 60000
Default:
current_speed_time_interval_msecs=1000
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
[mirrors].benchmark_oblivion
SYNOPSIS: use_benchmark_stats=0 | 1
If use_benchmark_stats=1 statistics on mirrors is used to rate them
and therefore improve performance.
Each time connection from a particular mirror closes mirror->dld_time,
and mirror->dld_size get increased (in case of unsuccessful connection
only time gets increased), so avg speed for a mirror can be
calculated:
mirror->avg_speed=mirror->dld_size/mirror->dld_time. (1)
When new segment is going to be started segget goes through the list of
the mirrors distfile/segment has, and asks each mirror for self_rating:
"ulong self_rating=mirror->mirror_on_the_wall();".
This way segget chooses a mirror with the best self_rating. To calculate
self_rating mirrors use the following formula:
self_rating=dld_time/dld_size*honesty. (2)
So mirrors actually say how bad they are.
Even mirrors can have critical times, so to give mirrors another chance
honesty was added to the formula (2). honesty can get values in
interval (0,1]. Each time connection from a mirror opens or closes mirror
sets its honesty=1. If mirror was asked for self_evaluation with
mirror->mirror_on_the_wall(), but wasn't chosen its honesty decreases
somewhat (see [mirror].benchmark_oblivion option), so next time it will
lie little bit more about how bad it's.
Default:
use_benchmark_results=1
***Note: at the moment use_benchmark_results can NOT be set to 0
BENCHMARK_OBLIVION
benchmark_oblivion option allows to adjust how fast segget "forgets"
benchmarking statistics on mirrors performance.
Each time mirror->mirror_on_the_wall() called, mirror decreases its
honesty (to have more chances next time) using the following formula:
honesty=honesty*100/(100+settings.benchmark_oblivion) (3)
Therefore, setting benchmark_oblivion=100 will make mirror look twice
less bad next time mirror->mirror_on_the_wall() called.
Default:
benchmark_oblivion=5
|
| |
|
|
|
|
|
|
| |
max_tries=10
ftp_response_timeout=180
timeout=500
max_connections_num_per_mirror=1
proxy_port=3128.
|
| |
|
|
|
|
|
|
| |
[mirrors].max_connections_num_per_mirror
Segget gathers stats on mirrors and uses it to rate them and therefore improve performance.
settings.forgetness option (by default set to 5) allows to adjust how fast segget "forgets"
about bad performance of a mirror.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
[connections].ftp_response_timeout, [connections].timeout.
Set the number of seconds to wait while trying to connect. Use 0 to wait
indefinitely. Pass a long. It should contain the maximum time in seconds that
you allow the connection to the server to take. This only limits the connection
phase, once it has connected, this option is of no more use. Set to zero to
disable connection timeout (it will then only timeout on the system's internal
timeouts). See also the TIMEOUT option.
default:
connection_timeout=15
Set a timeout period (in seconds) on the amount of time that the server is
allowed to take in order to generate a response message for a command before the
session is considered hung. While awaiting for a response, this value overrides
TIMEOUT. It is recommended that if used in conjunction with TIMEOUT, you set
FTP_RESPONSE_TIMEOUT to a value smaller than TIMEOUT.
default:
ftp_response_timeout=5000
maximum amount of time to download segment in seconds
Set the maximum number of seconds for a connection to execute.
Pass a long as parameter containing the maximum time in seconds that you allow
the transfer operation to take. Normally, name lookups can take a considerable
time and limiting operations to less than a few minutes risk aborting perfectly
normal operations.
default:
timeout=6000
|
| |
|
|
|
|
| |
Define maximum segment size in bytes
default:
max_segment_size=500000
|
| |
|
|
|
| |
Each distfile can only have up to max_connection_num_per_distfile
simultaneous connections.
|
| | |
|
| | |
|
| |
|
|
|
| |
checksum.cpp file contains functions for RMD160, SHA1 and SHA256 checksums
checks. libcrypto (libcrypto++) added to Makefile.
|
| |
|
|
| |
is less than MAX_CONNECTS.
|
| | |
|
| | |
|
| |
|
|
| |
Segments from now stored at ./tmp, distfiles at ./distfile dirs.
|
| | |
|
| | |
|
| |
|
|
| |
files
|
| | |
|
| | |
|
| | |
|
Kostyantyn Ovechko
mona
GitWeb