commit 6c8008eef7b9d58bbab4a3e2d89b9e159649f24e from: Jelmer Vernooij date: Sun Feb 05 23:03:55 2023 UTC Move swift README to contrib directory. commit - 4a97b84f8af18b4bc3393b41c5ab4e8384faa315 commit + 6c8008eef7b9d58bbab4a3e2d89b9e159649f24e blob - 4976b90a36c6fc3693937aa1aad00137c503c98b (mode 644) blob + /dev/null --- README.swift.rst +++ /dev/null @@ -1,133 +0,0 @@ -Openstack Swift as backend for Dulwich -====================================== -Fabien Boucher - -The module dulwich/contrib/swift.py implements dulwich.repo.BaseRepo -in order to being compatible with Openstack Swift. -We can then use Dulwich as server (Git server) and instead of using -a regular POSIX file system to store repository objects we use the -object storage Swift via its own API. - -c Git client <---> Dulwich server <---> Openstack Swift API - -This implementation is still a work in progress and we can say that -is a Beta version so you need to be prepared to find bugs. - -Configuration file ------------------- - -We need to provide some configuration values in order to let Dulwich -talk and authenticate against Swift. The following config file must -be used as template:: - - [swift] - # Authentication URL (Keystone or Swift) - auth_url = http://127.0.0.1:5000/v2.0 - # Authentication version to use - auth_ver = 2 - # The tenant and username separated by a semicolon - username = admin;admin - # The user password - password = pass - # The Object storage region to use (auth v2) (Default RegionOne) - region_name = RegionOne - # The Object storage endpoint URL to use (auth v2) (Default internalURL) - endpoint_type = internalURL - # Concurrency to use for parallel tasks (Default 10) - concurrency = 10 - # Size of the HTTP pool (Default 10) - http_pool_length = 10 - # Timeout delay for HTTP connections (Default 20) - http_timeout = 20 - # Chunk size to read from pack (Bytes) (Default 12228) - chunk_length = 12228 - # Cache size (MBytes) (Default 20) - cache_length = 20 - - -Note that for now we use the same tenant to perform the requests -against Swift. Therefore there is only one Swift account used -for storing repositories. Each repository will be contained in -a Swift container. - -How to start unittest ---------------------- - -There is no need to have a Swift cluster running to run the unitests. -Just run the following command in the Dulwich source directory:: - - $ PYTHONPATH=. python -m dulwich.contrib.test_swift - -How to start functional tests ------------------------------ - -We provide some basic tests to perform smoke tests against a real Swift -cluster. To run those functional tests you need a properly configured -configuration file. The tests can be run as follow:: - - $ DULWICH_SWIFT_CFG=/etc/swift-dul.conf PYTHONPATH=. python -m dulwich.contrib.test_swift_smoke - -How to install --------------- - -Install the Dulwich library via the setup.py. The dependencies will be -automatically retrieved from pypi:: - - $ python ./setup.py install - -How to run the server ---------------------- - -Start the server using the following command:: - - $ python -m dulwich.contrib.swift daemon -c /etc/swift-dul.conf -l 127.0.0.1 - -Note that a lot of request will be performed against the Swift -cluster so it is better to start the Dulwich server as close -as possible of the Swift proxy. The best solution is to run -the server on the Swift proxy node to reduce the latency. - -How to use ----------- - -Once you have validated that the functional tests is working as expected and -the server is running we can init a bare repository. Run this -command with the name of the repository to create:: - - $ python -m dulwich.contrib.swift init -c /etc/swift-dul.conf edeploy - -The repository name will be the container that will contain all the Git -objects for the repository. Then standard c Git client can be used to -perform operations against this repository. - -As an example we can clone the previously empty bare repository:: - - $ git clone git://localhost/edeploy - -Then push an existing project in it:: - - $ git clone https://github.com/enovance/edeploy.git edeployclone - $ cd edeployclone - $ git remote add alt git://localhost/edeploy - $ git push alt master - $ git ls-remote alt - 9dc50a9a9bff1e232a74e365707f22a62492183e HEAD - 9dc50a9a9bff1e232a74e365707f22a62492183e refs/heads/master - -The other Git commands can be used the way you do usually against -a regular repository. - -Note the daemon subcommands starts a Git server listening for the -Git protocol. Therefore there is no authentication or encryption -at all between the cGIT client and the GIT server (Dulwich). - -Note on the .info file for pack object --------------------------------------- - -The Swift interface of Dulwich relies only on the pack format -to store Git objects. Instead of using only an index (pack-sha.idx) -along with the pack, we add a second file (pack-sha.info). This file -is automatically created when a client pushes some references on the -repository. The purpose of this file is to speed up pack creation -server side when a client fetches some references. Currently this -.info format is not optimized and may change in future. blob - /dev/null blob + 4976b90a36c6fc3693937aa1aad00137c503c98b (mode 644) --- /dev/null +++ dulwich/contrib/README.swift.rst @@ -0,0 +1,133 @@ +Openstack Swift as backend for Dulwich +====================================== +Fabien Boucher + +The module dulwich/contrib/swift.py implements dulwich.repo.BaseRepo +in order to being compatible with Openstack Swift. +We can then use Dulwich as server (Git server) and instead of using +a regular POSIX file system to store repository objects we use the +object storage Swift via its own API. + +c Git client <---> Dulwich server <---> Openstack Swift API + +This implementation is still a work in progress and we can say that +is a Beta version so you need to be prepared to find bugs. + +Configuration file +------------------ + +We need to provide some configuration values in order to let Dulwich +talk and authenticate against Swift. The following config file must +be used as template:: + + [swift] + # Authentication URL (Keystone or Swift) + auth_url = http://127.0.0.1:5000/v2.0 + # Authentication version to use + auth_ver = 2 + # The tenant and username separated by a semicolon + username = admin;admin + # The user password + password = pass + # The Object storage region to use (auth v2) (Default RegionOne) + region_name = RegionOne + # The Object storage endpoint URL to use (auth v2) (Default internalURL) + endpoint_type = internalURL + # Concurrency to use for parallel tasks (Default 10) + concurrency = 10 + # Size of the HTTP pool (Default 10) + http_pool_length = 10 + # Timeout delay for HTTP connections (Default 20) + http_timeout = 20 + # Chunk size to read from pack (Bytes) (Default 12228) + chunk_length = 12228 + # Cache size (MBytes) (Default 20) + cache_length = 20 + + +Note that for now we use the same tenant to perform the requests +against Swift. Therefore there is only one Swift account used +for storing repositories. Each repository will be contained in +a Swift container. + +How to start unittest +--------------------- + +There is no need to have a Swift cluster running to run the unitests. +Just run the following command in the Dulwich source directory:: + + $ PYTHONPATH=. python -m dulwich.contrib.test_swift + +How to start functional tests +----------------------------- + +We provide some basic tests to perform smoke tests against a real Swift +cluster. To run those functional tests you need a properly configured +configuration file. The tests can be run as follow:: + + $ DULWICH_SWIFT_CFG=/etc/swift-dul.conf PYTHONPATH=. python -m dulwich.contrib.test_swift_smoke + +How to install +-------------- + +Install the Dulwich library via the setup.py. The dependencies will be +automatically retrieved from pypi:: + + $ python ./setup.py install + +How to run the server +--------------------- + +Start the server using the following command:: + + $ python -m dulwich.contrib.swift daemon -c /etc/swift-dul.conf -l 127.0.0.1 + +Note that a lot of request will be performed against the Swift +cluster so it is better to start the Dulwich server as close +as possible of the Swift proxy. The best solution is to run +the server on the Swift proxy node to reduce the latency. + +How to use +---------- + +Once you have validated that the functional tests is working as expected and +the server is running we can init a bare repository. Run this +command with the name of the repository to create:: + + $ python -m dulwich.contrib.swift init -c /etc/swift-dul.conf edeploy + +The repository name will be the container that will contain all the Git +objects for the repository. Then standard c Git client can be used to +perform operations against this repository. + +As an example we can clone the previously empty bare repository:: + + $ git clone git://localhost/edeploy + +Then push an existing project in it:: + + $ git clone https://github.com/enovance/edeploy.git edeployclone + $ cd edeployclone + $ git remote add alt git://localhost/edeploy + $ git push alt master + $ git ls-remote alt + 9dc50a9a9bff1e232a74e365707f22a62492183e HEAD + 9dc50a9a9bff1e232a74e365707f22a62492183e refs/heads/master + +The other Git commands can be used the way you do usually against +a regular repository. + +Note the daemon subcommands starts a Git server listening for the +Git protocol. Therefore there is no authentication or encryption +at all between the cGIT client and the GIT server (Dulwich). + +Note on the .info file for pack object +-------------------------------------- + +The Swift interface of Dulwich relies only on the pack format +to store Git objects. Instead of using only an index (pack-sha.idx) +along with the pack, we add a second file (pack-sha.info). This file +is automatically created when a client pushes some references on the +repository. The purpose of this file is to speed up pack creation +server side when a client fetches some references. Currently this +.info format is not optimized and may change in future.