Gemdeps Tweaks tracker
Created by: Balasankar C
Issue to report and track exceptions and other tweaks needed for gemdeps and progressbars.
Tasks:
-
Add version mismatch check between an app's requirement and Debian packaged version -
Add comparison between version requirements of gems common for different apps -
Add API for future expansions -
Handle multiple gems depending different versions of same gem -
Check dependencies of exact version for second level dependencies ('fog 1.25' instead of 'fog latest') - Blocked by issue -
Allow sorting option to be passed as url parameter (like sort=satisfied) -
Allow checking compatibility for all reverse dependencies (for example does updating sass-rails from 5.0.3 to 5.0.4 break any?) -
Use Gemfile.lock for better handling of dependencies. -
Split incomplete markdown to categories like here (unpackaged, major, minor etc) -
check gitlab-shell and gitlab-workhorse (these are specified by just text files) -
Generate list of dependencies removed since last version - Idea from Stan Hu (GitLab)
Imported comments:
By Balasankar C on 2016-06-18T17:00:13.526Z
Marked the task Use Gemfile.lock for better handling of dependencies. as completed
By Balasankar C on 2016-06-18T17:00:13.512Z
Marked the task Check dependencies of exact version for second level dependencies ('fog 1.25' instead of 'fog latest') as completed
By Praveen Arimbrathodiyil on 2016-06-18T16:16:45.471Z
@balasankarc looks great! Thanks :) gitlab_omniauth-ldap should be ignored as well.
By Balasankar C on 2016-06-18T08:24:06.316Z
@pravi I have tried to rewrite the tracker to go along with out above conversation (new API, unsatisfied gems and dependencies only etc etc) and generated a status bar for current develop branch of Gitlab. It can be found here - http://debian.fosscommunity.in/status/?appname=gitlab-new . It would be great if someone could verify whether it shows the correct information.
@rahulkrishnanfs @sudheeshshetty I believe I have restructured the tracker code to be more modular, less complex, more readable and better documented. If you wanna hack on it, I think this change will be useful. Check the merge request mentioned in first line.
PS : I will fix rugged's status soon.
By Praveen Arimbrathodiyil on 2016-06-11T11:38:18.251Z
@balasankarc rugged is satisfied. 024b should be 0.24~b.
By Praveen Arimbrathodiyil on 2016-06-11T11:22:32.171Z
@balasankarc its a good plan.
For 2.2, option 1, if we can look in Gemfile.lock for the version gitlab is using, that should be preferred (its tested by them).
Or option 2, we can go for the maximum version that satisfies (we can avoid frequent updates).
If we have run the checks for a particular version, we can cache it and reuse it I guess.
By Balasankar C on 2016-06-11T11:00:52.761Z
@pravi I have been thinking that for some time. Since we already have them running, we need not maintain status of every gem.
1. Take a gem
2. If it is packaged in Debian, then
2.1 If the version satisfies the requirement, continue to next gem. We need not care about its dependencies. It may already have been satisfied
2.2 Else, get the minimum version that satisfies the requirement and find its dependencies
3. Else, find the minimum version that satisfies the requirement and find its dependencies
There are many gems that remain without any change since we started doing this. Let us consider them only when the requirement change.
By Balasankar C on 2016-06-11T10:45:27.573Z
@pravi We can use this API : http://rubygems.org/api/v1/dependencies.json?gems=rexical
We have to discuss and finalize on which version we should consider for finding dependencies in the following cases
- Exact version specified - We have no option, just find out its dependencies
- ~> relation - We have a minimum and maximum limit. We will choose the smallest satisfiable one
-
= relation - We have a
maximumminimum limit. We will choose the smallest satisfiable one. - No version is specified - What should we do?
@rahulkrishnanfs You will have to write methods for getting list of versions that will satisfy the requirement and finding the smallest one from them. Also, you will have to change the current code to use the new API.
Code is available at https://gitlab.com/balasankarc/python-gemdeps/
By Praveen Arimbrathodiyil on 2016-06-11T10:54:05.281Z
@balasankarc for option 2 and 3, see if debian already have a version in that range. 3, >= means no maximum limit, only minimum. 4, no version means any version, see if debian has any version of it.
Btw @sudheeshshetty you were also asking about source for gemdeps.
By Praveen Arimbrathodiyil on 2016-06-01T12:49:24.781Z
@balasankarc yes, runtime is enough for us.
rugged version requirement is >= 0.24b, it should be considered as 0.24~b when comparing.
By Balasankar C on 2016-04-27T05:58:12.424Z
Yay!! http://bundler.rubygems.org/ will give us the necessary information that we need, but only for runtime dependencies (Thanks to Sreehari for finding it out)
@pravi Can we live with runtime dependencies alone?
By Praveen Arimbrathodiyil on 2016-04-03T17:28:32.571Z
@balasankarc ok
By Balasankar C on 2016-04-02T15:02:22.812Z
@pravi : I can't understand the structure of Gemfile.lock. I can find out the exact versions to be used from Gemfile.lock . But I can't get the requirements (that is >=, ~> type relations) that they finally chose.
If that requires way too much coding, I will skip that and wait for Srihari to close that bug.
By Balasankar C on 2016-04-01T18:35:57.897Z
@pravi : I will get into that soon. Need to sleep now.
By Praveen Arimbrathodiyil on 2016-04-01T18:32:39.848Z
@balasankarc do you check for exact versions too? ie, will it show latest version at rubygems.for even if Gemfile.lock lists an older version?
By Balasankar C on 2016-04-01T17:45:58.199Z
@pravi : I temporarily fixed it. Gems not present in Gemfile.lock will not be displayed in status bar. :)
By Praveen Arimbrathodiyil on 2016-03-13T05:44:46.920Z
@balasankarc can you use Gemfile.lock to find exact versions of dependencies while we wait for rubygems.org to implement this option?
By Praveen Arimbrathodiyil on 2016-02-14T02:52:31.093Z
@balasankarc can you check if required version is already in jessie when generating sequencial list?
Also add ruby-test-unit, ruby-rspec and gem2deb at first.
By Balasankar C on 2016-01-17T08:14:13.279Z
@pravi : Fixed everything. Also added links to ITPs/ RFPs/ tracker.debian.org etc depending on package status. :)
By Balasankar C on 2016-01-17T06:34:58.813Z
Great. Lemme fix that (Problems of not knowing Ruby and what all can come in a Gemfile).
:)
By Praveen Arimbrathodiyil on 2016-01-17T06:26:13.298Z
@balasankarc its a bug in your parser, platform is not a gem :(
group :metrics do
gem 'allocations', '~> 1.0', require: false, platform: :mri
gem 'method_source', '~> 0.8', require: false
gem 'influxdb', '~> 0.2', require: false
gem 'connection_pool', '~> 2.0', require: false
end
By Balasankar C on 2016-01-17T06:21:56.841Z
aasm, acts_as_tree_rails3, tr8n, will_filter where are these coming from? I did not need them for gitlab 8.4.0. I can't find them in Gemfile.lock as well.
aasm > platform > gitlab (metrics). You asked me to enable metrics group in statusbar na? The others are probably from the same group.
I will add exception to cause.
By Praveen Arimbrathodiyil on 2016-01-17T06:15:26.720Z
Also you can ignore cause, we don't need it (needed only for ruby < 2.1.0)
By Praveen Arimbrathodiyil on 2016-01-17T06:14:33.796Z
@balasankarc aasm, acts_as_tree_rails3, tr8n, will_filter where are these coming from? I did not need them for gitlab 8.4.0. I can't find them in Gemfile.lock as well.
By Praveen Arimbrathodiyil on 2016-01-12T05:45:12.557Z
On demand, means when we need them, like this case of sass-rails. So we add rails_admin json for sass-rails, may be another one for tilt or net-ssh etc.
By Praveen Arimbrathodiyil on 2016-01-12T05:43:07.864Z
Alright. But can we add more pre-compiled json files on demand? The big ones, maybe having more than 5 dependencies like rails_admin.
Well, even if the declared dependencies are not correct, we have to patch them (relax deps) or diaspora, gitlab etc will fail on 'bundle install --local'
By Balasankar C on 2016-01-12T05:31:31.081Z
@pravi Gemdeps currently works based on precompiled JSON files (generated daily using a cron job). What you are suggesting would require
- running gemdeps on all reverse dependencies (to find out version requirements of their dependendencies, coz neither
apt-cache rdepends
norapt-rdepends -r
provide that information) and get their version requirements - store them as JSON
- implement another checker that uses is JSON, like we do for comparator
Steps 1 itself make it not feasible to integrate this in gemdeps. We will have to generate metadata about a significant part of the entire Debian Ruby ecosystem.
Theoretically, I think it is possible, but I am strongly against implementing it (not generally, but specifically against integrating it with gemdeps). One thing we can do is try to extend apt-cache rdepends
to list out the version specified in debian control also. Then it is a matter of a small script to perform the comparison, which can be done locally. Let us go in that direction (Or is it already implemented and I missed it?).
PS : Checking for breakage must be done manually coz chances of false positives are greater in version requirements (many of the gems simply specify version dependency when the gems simply work with any version of the dependencies).
By Praveen Arimbrathodiyil on 2016-01-12T05:01:20.639Z
We should be able check compatibility with all reverse dependencies. For example when sass-rails is updated from 5.0.3 to 5.0.4, currently we can check if diaspora and gitlab will break or not, but we should be able to see other reverse dependency status too.
$ apt-cache rdepends ruby-sass-rails
ruby-sass-rails
Reverse Depends:
ruby-sass
ruby-bootstrap-sass
|ruby-compass-rails
ruby-rails-admin
ruby-rails-admin
ruby-sass
|ruby-compass-rails
diaspora
diaspora
gitlab
diaspora
ruby-rails-admin
ruby-rails-admin
|ruby-compass-rails
ruby-bootstrap-sass
ruby-rails
diaspora
By Praveen Arimbrathodiyil on 2016-01-11T19:13:29.807Z
Marked the task Check dependencies of exact version for second level dependencies ('fog 1.25' instead of 'fog latest') as incomplete
By Praveen Arimbrathodiyil on 2016-01-11T19:13:28.448Z
Marked the task Check dependencies of exact version for second level dependencies ('fog 1.25' instead of 'fog latest') as completed
By Balasankar C on 2016-01-07T04:44:23.924Z
Should we do that? At any point of time, we will be ideally dealing with only two versions. The latest one (for experimental) and the one before that (for Unstable). I think the current setup is sufficient for that. Also like you said, compare will be damn dirty.
By Praveen Arimbrathodiyil on 2016-01-02T20:07:38.505Z
how about making version also a parameter? like appname=gitlab&version=8.2.1 is more readable I think. Though it may make compare complicated.
By Balasankar C on 2016-01-01T17:31:56.882Z
Someone is implementing a subset of what I need in rubygems. Asked him to see if he can do what I need also.
https://github.com/rubygems/rubygems.org/pull/1154
By Praveen Arimbrathodiyil on 2016-01-01T13:47:34.858Z
I think tracker showing celluloid, celluloid-io is wrong. eye needs ~> 0.16.0 and it cannot be updated to 0.17.2 without update eye.
By Balasankar C on 2016-01-01T08:30:55.310Z
Marked the task Allow sorting option to be passed as url parameter (like sort=satisfied) as completed
By Balasankar C on 2016-01-01T08:30:38.251Z
we need to be able pass sorting option to the tracker as well.
@pravi Done.
Self Note: I seriously need to learn JavaScript in a better, organized way than these random hacks. :P
By Praveen Arimbrathodiyil on 2016-01-01T06:16:50.033Z
We need to special handle [show some kind of warning] rails-assets-jquery (1.11.3 is fine), rails-timeago (https://github.com/diaspora/diaspora/issues/3855).
and rails-assets-diaspora_jsxc is satisfied.
diaspora-vines can be ignored for now until chat is declared stable.
By Praveen Arimbrathodiyil on 2016-01-01T06:01:13.460Z
mysql2 can be ignored as we need only pg for now
By Praveen Arimbrathodiyil on 2016-01-01T06:00:21.564Z
@balasankarc we need to be able pass sorting option to the tracker as well. For example, when we want to update diaspora to 0.5.5, we want to know which gems are not satisfied and so giving a direct link to sorted by satisfied makes sense, rather than tell people to sort by satisfied and look at a particular gem serial number.
By Balasankar C on 2015-12-30T18:01:45.312Z
Marked the task Handle multiple gems depending different versions of same gem as completed
By Balasankar C on 2015-12-30T18:01:19.443Z
omniauth-oauth is shown as ~>`1.0 and satisfied but omniauth-twitter needs ~> 1.1 (omniauth-tumblr and omniauth-bitbucket needs ~> 1.0 only)
@pravi This is done by balasankarc/python-gemdeps@7b1e09e099d19e8772a8d7ca0e4de84bbdb677d8
By Praveen Arimbrathodiyil on 2015-12-28T20:09:39.887Z
gitlab_meta can be ignored
By Praveen Arimbrathodiyil on 2015-12-27T14:19:57.387Z
It should be satisfied in experimental.
$ apt-cache policy ruby-rails-assets-diaspora-jsxc
ruby-rails-assets-diaspora-jsxc:
Installed: 0.1.4+dfsg~alpha.1-1
Candidate: 0.1.4+dfsg~alpha.1-1
Version table:
0.1.4+dfsg~alpha.1-1 0
1 http://cdn.debian.net/debian/ experimental/main amd64 Packages
*** 0.1.4+dfsg~alpha.1-1 0
100 /var/lib/dpkg/status
0.1.1~dfsg-1 0
500 http://debian.sil.at/debian/ sid/main amd64 Packages
By Balasankar C on 2015-12-27T13:38:58.905Z
rails-assets-diaspora_jsxc checking is not correct. It should be satisfied no.
I am seeing it as satisfied. Try clearing your cache.
By Balasankar C on 2015-12-27T13:36:32.743Z
omniauth-oauth is shown as ~>`1.0 and satisfied but omniauth-twitter needs ~> 1.1 (omniauth-tumblr and omniauth-bitbucket needs ~> 1.0 only)
I had anticipated this will happen. I wrote the code to skip duplicate dependencies. We need an additional check to select the highest requirement. Was too lazy to fix it when I wrote the code initially. I will do it soon.
And you are missing the whole production section (rails_admin, eye etc)
Yes. I was doing only the group 'runtime'. This, however, is easy to fix and I'll push it soon.
By Praveen Arimbrathodiyil on 2015-12-27T12:00:21.242Z
And you are missing the whole production section (rails_admin, eye etc)
By Praveen Arimbrathodiyil on 2015-12-27T11:14:24.376Z
rails-assets-diaspora_jsxc checking is not correct. It should be satisfied no.
By Praveen Arimbrathodiyil on 2015-12-27T06:59:32.334Z
@balasankarc more exciting challenge (complexity) for you
omniauth-oauth is shown as ~>`1.0 and satisfied but omniauth-twitter needs ~> 1.1 (omniauth-tumblr and omniauth-bitbucket needs ~> 1.0 only)
By Balasankar C on 2015-12-26T09:45:21.498Z
@pravi Ok. I've started another run of gemdeps. Will get reflected within 20 minutes.
By Praveen Arimbrathodiyil on 2015-12-26T09:39:31.400Z
@balasankarc can you change 0.5.3.0 to 0.5.4.0 (as we want to move 0.5.4.0 and not 0.5.3.0 from experimental to unstable)
By Balasankar C on 2015-12-24T18:27:59.128Z
Added basic API to gemdeps. http://debian.fosscommunity.in/api/
By Balasankar C on 2015-12-24T18:27:27.486Z
gollum-grit_adapter needs exception, we use gollum-rugged_adapter instead.
Done
By Praveen Arimbrathodiyil on 2015-12-23T18:18:08.548Z
gollum-grit_adapter needs exception, we use gollum-rugged_adapter instead.
By Balasankar C on 2015-12-23T07:01:32.841Z
Title changed from Add dependency and debian version comparison to Gemdeps to Gemdeps Tweaks tracker
By Balasankar C on 2015-12-23T07:00:30.372Z
eco, rb-fsevent, eco-source can be ignored.
Done
ruby-rails-assets-jeresig-jquery.hotkeys, ruby-rails-assets-jakobmattsson-jquery-elastic, ruby-oembed in the archive and needs exceptions.
Done
messagebus_ruby_api needs an exception for 1.0.3 version.
Done
also specify timezone in timestamp
Done
ruby-fogbugz needs an exception
Done
By Praveen Arimbrathodiyil on 2015-12-22T06:04:22.326Z
ruby-fogbugz needs an exception
By Praveen Arimbrathodiyil on 2015-12-22T06:02:49.797Z
@balasankarc also specify timezone in timestamp
By Praveen Arimbrathodiyil on 2015-12-22T05:56:13.502Z
messagebus_ruby_api needs an exception for 1.0.3 version.
By Praveen Arimbrathodiyil on 2015-12-22T03:56:42.052Z
eco, rb-fsevent, eco-source can be ignored.
ruby-rails-assets-jeresig-jquery.hotkeys, ruby-rails-assets-jakobmattsson-jquery-elastic, ruby-oembed in the archive and needs exceptions.
By Balasankar C on 2015-12-20T09:49:45.322Z
when unicorn 4.9.0 was in experimental. It shown not satisfied and suite unstable. Only new packages not yet having an older version in unstable are shown as suite experimental. When a package is available in unstable and experimental, it only shows as unstable.
I know. I fixed it and at the same time you uploaded it to Unstable. :D . Check how ruby-httpclient is now shown as satisfied, but is in experimental.
If diaspora update dependency of bootstrap-sass to newer versions, we need to know. So the exception should be only for 2.3.2.2 unlike mini_portail (which we always ignore).
Ok. Will do that. (We are having pretty weird exceptions. ;) )
By Praveen Arimbrathodiyil on 2015-12-20T09:27:03.439Z
@balasankarc when unicorn 4.9.0 was in experimental. It shown not satisfied and suite unstable. Only new packages not yet having an older version in unstable are shown as suite experimental. When a package is available in unstable and experimental, it only shows as unstable.
If diaspora update dependency of bootstrap-sass to newer versions, we need to know. So the exception should be only for 2.3.2.2 unlike mini_portail (which we always ignore).
By Balasankar C on 2015-12-20T08:51:50.890Z
@pravi I fixed it. We don't want a new color. We just want to sort based on 'suite'. The ones falling under Experimental are the ones we need. We simple have to handle them.
About bootstrap-sass, since it is embedded, we need not display it in the progressbar, like unwanted stuff like mini_portile and newrelic-*. Right?
By Balasankar C on 2015-12-20T07:08:56.965Z
@pravi Ok. Suggest me a color.
By Praveen Arimbrathodiyil on 2015-12-20T07:08:35.016Z
bootstrap-sass 2.3.2.2 version is embedded inside diaspora and we can add an exception for that particular version.
By Praveen Arimbrathodiyil on 2015-12-19T06:20:42.421Z
@balasankarc currently it does not specify if a version in experimental satisfies the requirement or not (for example unicorn 4.9.0 in experimental satisfies diaspora requirement but it is shown as "Satisfied: No"). May be use a different colour for those are satisfied in experimental.
By Balasankar C on 2015-12-14T18:54:46.355Z
Basic work implemented in http://debian.fosscommunity.in