gitlab: not sending emails "can't modify frozen String"
GitLab is unable to pass email to MTA. Here is what Sidekiq is logging:
Aug 01 11:07:00 tst1inf26 gitlab-sidekiq[32685]: 2016-08-01T01:07:00.796Z 32685 TID-zjn94 NewNoteWorker JID-6559884e9f628531f076b083 INFO: start
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: 2016-08-01T01:07:01.125Z 32685 TID-1aulwg ActionMailer::DeliveryJob JID-ca44ffd38574fe740d4594fc INFO: start
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: 2016-08-01T01:07:01.321Z 32685 TID-1aulwg ActionMailer::DeliveryJob JID-ca44ffd38574fe740d4594fc INFO: fail: 0.196 sec
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: 2016-08-01T01:07:01.321Z 32685 TID-1aulwg WARN: {"class"=>"ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper", "wrapped"=>"ActionMailer::DeliveryJob", "queue"=>"mailers", "args"=>[{"job_class"=>"ActionMailer::DeliveryJob", "job_id"=>"bc48bbcc-ef5d-459b-9c98-d98e7811646b", "queue_name"=>"mailers", "arguments"=>["Notify", "note_issue_email", "deliver_now", 4, 42], "locale"=>"en"}], "retry"=>true, "jid"=>"ca44ffd38574fe740d4594fc", "created_at"=>1470013621.12361, "enqueued_at"=>1470013621.1237197, "error_message"=>"can't modify frozen String", "error_class"=>"RuntimeError", "failed_at"=>1470013621.318343, "retry_count"=>0}
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: 2016-08-01T01:07:01.321Z 32685 TID-1aulwg WARN: RuntimeError: can't modify frozen String
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: 2016-08-01T01:07:01.321Z 32685 TID-1aulwg WARN: /usr/lib/ruby/vendor_ruby/mail/utilities.rb:22:in `force_encoding'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/mail/utilities.rb:22:in `quote_phrase'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/mail/elements/address.rb:51:in `format'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/share/gitlab/app/mailers/base_mailer.rb:8:in `block in <class:BaseMailer>'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/action_mailer/base.rb:811:in `instance_eval'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/action_mailer/base.rb:811:in `block in mail'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/action_mailer/base.rb:810:in `each'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/action_mailer/base.rb:810:in `mail'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/share/gitlab/app/mailers/notify.rb:124:in `mail_thread'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/share/gitlab/app/mailers/notify.rb:152:in `mail_answer_thread'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/share/gitlab/app/mailers/emails/notes.rb:20:in `note_issue_email'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/abstract_controller/base.rb:198:in `process_action'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/abstract_controller/callbacks.rb:20:in `block in process_action'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/active_support/callbacks.rb:88:in `__run_callbacks__'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/active_support/callbacks.rb:778:in `_run_process_action_callbacks'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/active_support/callbacks.rb:81:in `run_callbacks'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/abstract_controller/callbacks.rb:19:in `process_action'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/abstract_controller/base.rb:137:in `process'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/action_view/rendering.rb:30:in `process'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/action_mailer/base.rb:596:in `block in process'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/active_support/notifications.rb:164:in `block in instrument'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/active_support/notifications/instrumenter.rb:20:in `instrument'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/active_support/notifications.rb:164:in `instrument'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/action_mailer/base.rb:593:in `process'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/action_mailer/base.rb:584:in `initialize'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/action_mailer/message_delivery.rb:25:in `new'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/action_mailer/message_delivery.rb:25:in `__getobj__'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/action_mailer/message_delivery.rb:34:in `message'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/action_mailer/message_delivery.rb:85:in `deliver_now'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/action_mailer/delivery_job.rb:10:in `perform'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/active_job/execution.rb:32:in `block in perform_now'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/active_support/callbacks.rb:117:in `call'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/active_support/callbacks.rb:555:in `block (2 levels) in compile'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/active_support/callbacks.rb:505:in `call'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/active_support/callbacks.rb:498:in `block (2 levels) in around'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/active_support/callbacks.rb:343:in `block (2 levels) in simple'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/i18n.rb:257:in `with_locale'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/active_job/translation.rb:7:in `block (2 levels) in <module:Translation>'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/active_support/callbacks.rb:441:in `instance_exec'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/active_support/callbacks.rb:441:in `block in make_lambda'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/active_support/callbacks.rb:342:in `block in simple'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/active_support/callbacks.rb:497:in `block in around'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/active_support/callbacks.rb:505:in `call'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/active_support/callbacks.rb:498:in `block (2 levels) in around'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/active_support/callbacks.rb:343:in `block (2 levels) in simple'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/active_job/logging.rb:23:in `block (4 levels) in <module:Logging>'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/active_support/notifications.rb:164:in `block in instrument'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/active_support/notifications/instrumenter.rb:20:in `instrument'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/active_support/notifications.rb:164:in `instrument'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/active_job/logging.rb:22:in `block (3 levels) in <module:Logging>'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/active_job/logging.rb:43:in `block in tag_logger'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/active_support/tagged_logging.rb:68:in `block in tagged'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/active_support/tagged_logging.rb:26:in `tagged'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/active_support/tagged_logging.rb:68:in `tagged'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/active_job/logging.rb:43:in `tag_logger'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/active_job/logging.rb:19:in `block (2 levels) in <module:Logging>'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/active_support/callbacks.rb:441:in `instance_exec'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/active_support/callbacks.rb:441:in `block in make_lambda'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/active_support/callbacks.rb:342:in `block in simple'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/active_support/callbacks.rb:497:in `block in around'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/active_support/callbacks.rb:505:in `call'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/active_support/callbacks.rb:92:in `__run_callbacks__'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/active_support/callbacks.rb:778:in `_run_perform_callbacks'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/active_support/callbacks.rb:81:in `run_callbacks'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/active_job/execution.rb:31:in `perform_now'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/active_job/execution.rb:21:in `execute'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/active_job/queue_adapters/sidekiq_adapter.rb:42:in `perform'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/sidekiq/processor.rb:150:in `execute_job'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/sidekiq/processor.rb:132:in `block (2 levels) in process'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/sidekiq/middleware/chain.rb:127:in `block in invoke'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/sidekiq/middleware/server/active_record.rb:6:in `call'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/sidekiq/middleware/chain.rb:129:in `block in invoke'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/sidekiq/middleware/server/retry_jobs.rb:74:in `call'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/sidekiq/middleware/chain.rb:129:in `block in invoke'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/sidekiq/middleware/server/logging.rb:11:in `block in call'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/sidekiq/logging.rb:30:in `with_context'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/sidekiq/middleware/server/logging.rb:7:in `call'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/sidekiq/middleware/chain.rb:129:in `block in invoke'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/sidekiq/middleware/chain.rb:132:in `invoke'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/sidekiq/processor.rb:127:in `block in process'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/sidekiq/processor.rb:166:in `stats'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/sidekiq/processor.rb:126:in `process'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/sidekiq/processor.rb:79:in `process_one'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/sidekiq/processor.rb:67:in `run'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/sidekiq/util.rb:16:in `watchdog'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: /usr/lib/ruby/vendor_ruby/sidekiq/util.rb:24:in `block in safe_thread'
Aug 01 11:07:01 tst1inf26 gitlab-sidekiq[32685]: 2016-08-01T01:07:01.356Z 32685 TID-zjn94 NewNoteWorker JID-6559884e9f628531f076b083 INFO: done: 0.56 sec
@praveen, I hope you have any ideas...