codedecoder

breaking into the unknown…


Leave a comment

simple file upload in rails

I have been doing file upload in rails using attachment_fu in initial days and then paperclip since its arrival. But have never used simple file upload i,e without using any gems. Recently I have to redesign the paperclip upload functionality of one of my application(will discuss in some other post), So thought of starting from beginning i,e how simple uploader works, what benefits paperclip or other available gem adds to it etc .

I am assuming you already have some rails 4 application running. We will add photo upload feature to it. This will be our flow.

->There will be a upload button on new photo page.
->User upload a pic using the simple uploaded.
-> On successful upload he will be redirected to show page, displaying the uploaded image.

generate the model and controller with below command on your project root from terminal

$ rails g model Photo pic:attachement
invoke active_record
create db/migrate/20150626122526_create_photos.rb
create app/models/photo.rb

If you see the generated migration file, it has below content

class CreatePhotos < ActiveRecord::Migration
  def change
    create_table :photos do |t|
      t.attachment :pic
      t.timestamps null: false
    end
  end
end

Run migration to add the photos table

$rake db:migrate

Once the migration is complete and you chek the schema.rb file, you will see below detail of the photos table.

  create_table "photos", force: :cascade do |t|
    t.string   "pic_file_name"
    t.string   "pic_content_type"
    t.integer  "pic_file_size"
    t.datetime "pic_updated_at"
    t.datetime "created_at",       null: false
    t.datetime "updated_at",       null: false
  end

So you can see that t.attachement in the migration file get converted into below 4 fields.

t.string “pic_file_name”
t.string “pic_content_type”
t.integer “pic_file_size”
t.datetime “pic_updated_at”

They represent internal property of any attachement. When user upload any file, you need to read these values from params and need to update these fields. Let generate the controller now

$ rails g controller photos new show create.

It will generate new, show and create action in photos controller and the corresponding views.
It will also generate get routes for all these actions.

  get 'photos/new'
  get 'photos/create'
  get 'photos/show'

remove them and replace with resourceful routes.

resources :photos

write the file upload code to the photos/new.html.erb

<%= form_for @photo, :url => photos_path, :html => { :multipart => true } do |form| %>
  <%= form.file_field :pic %>
  <%=form.submit 'upload'%>
<% end %>

This simply provide a file upload field with a upload button, which submit data to create action.

The whole controller code look as below:

class PhotosController < ApplicationController
  before_action :set_photo, only: [:show]

  def show
  end

  def new
    @photo = Photo.new
  end

  def create
    @photo = Photo.new
    @photo_object = params[:photo][:pic]
    @photo.pic_file_name = @photo_object.original_filename
    @photo.pic_content_type = @photo_object.content_type
    respond_to do |format|
      if @photo.save
        file_name_filesystem = @photo.id.to_s + @photo_object.original_filename
        photo_path_on_filesystem = Rails.root.join('public','uploads', file_name_filesystem)
        File.open(photo_path_on_filesystem,'wb') do |file| 
          file.write(@photo_object.read)
        end
        format.html { redirect_to @photo, notice: 'Photo was successfully created.' }
        format.json { render :show, status: :created, location: @photo }
      else
        format.html { render :new }
        format.json { render json: @photo.errors, status: :unprocessable_entity }
      end
    end
  end

  private
  def set_photo
    @photo = Photo.find(params[:id])
  end
end

The points to notice here is the create action. If you put debugger(I use byebug) inside the create action and try to see the params, it look as below

(byebug) pp params[:photo][:pic]
#<ActionDispatch::Http::UploadedFile:0xb3b9be60
@content_type=”image/jpeg”,
@headers=
“Content-Disposition: form-data; name=\”photo[pic]\”; filename=\”user1.jpeg\”\r\nContent-Type: image/jpeg\r\n”,
@original_filename=”user1.jpeg”,
@tempfile=#<File:/tmp/RackMultipart20150626-7518-liirs5.jpeg>>

(byebug) pp params[:photo][:pic].content_type
“image/jpeg”

(byebug) pp params[:photo][:pic].original_filename
“user1.jpeg”

(byebug) pp params[:photo][:pic].tempfile
#<File:/tmp/RackMultipart20150626-7518-liirs5.jpeg>

(byebug) pp params[:photo][:pic].tempfile.path
“/tmp/RackMultipart20150626-7518-liirs5.jpeg”

So you can see that params[:photo][:pic] is an object in itself from which we can get the name of the file, its content etc.

We have first saved file_name and content_type in db, and then write the files content in public/uploads directory of our project. So that when user visit show page next time we will show him the uploaded image.

NOTE: Instead of writing the content to filesystem, you can send it to any API which handle its storage or do whatever with the data you want to do.

Now in show.html.erb we have below code to render the uploaded photo.

 
<% pic_path = @photo.id.to_s + @photo.pic_file_name%> 
<%img src="/uploads/<%=pic_path%>" alt="" />

So from above implementation, now we can see what paperclip or other gem add to it:

-> You do not need to handle writing the uplaoded file to filesystem yourself. Gem will take care of it
-> Gem can handle storing the uploaded file on ThirdParty storage like amazone
-> They can provide further processing on image like creating tumbnail, applying affects etc.

Advertisements


Leave a comment

heroku timeout code H13

Recently while going live with one of our financial domain project on Heroku we get with this dreaded error.

at=error code=H13 desc=”Connection closed without response”

The error occurring whenever a user try to submit his loan application. Basically a user never able to submit his application due to the above error as the process getting killed due to response taking too long to comeback.

some of the culprit line in logs look like as below:

2015-06-08T19:10:34.297539+00:00 heroku[router]: at=error code=H13 desc=”Connection closed without response” method=POST path=”/loans/credit_request” host=getfinance.com request_id=0f4a7e2f-28a6-4542-a308-db3a9a72c0e6 fwd=”90.161.130.26″ dyno=web.2 connect=0ms service=26007ms status=503 bytes=0
2015-06-08T19:10:34Z app[postgres.32439]: [MAUVE] could not receive data from client: Connection reset by peer
2015-06-08T19:10:34.277236+00:00 app[web.2]: E, [2015-06-08T19:10:34.277078 #3] ERROR — : worker=0 PID:9 timeout (26s > 25s), killing

So basically Heroku is not to blamed. It is right in killing a process if it see no response coming in 25 second.

We know that oue API is the culprit. This is how our whole application work

-> User come to our application build in rails
-> Fill the details related to loan he want to apply and submit it.user see progress bar rotating
-> On submit we capture the data and make a API call to another application written in java.
-> The API do a lot of processing on the data, create citadel certificate, register the user on OpenAm,trigger few emails etc and then send back response.
-> when response is success we reload the page with success message or render the error

It is the second last step in which Heroku seeing a problem. Heroku keep waiting for the response, and when it see that it is exceeding 25 second, it kill the existing process, so the user page never get refreshed and hanged for ever with the progress bar rotating.

So we know the problem…but what will be the solution.

Idle solution is to improve on our API code and make it to return a response in say 15 second(our taking between 25 to 30 seconds). but it is long term plan. we are already live and it is not that we have not worked on reducing the response time(well it is crime if you make user to wait 30 second for response) but since a lot of processing undergo before sending the response, we can’t do much.

So we decided to live with the current response time and fool Heroku to believe that request response cycle is working.

Here is the Plan:
-> push the API call to background job
-> submit the form as ajax
-> In the action which handle submit, trigger the background job and render the job_id , the path to which need to be redirected and other parameter you want to reuse
-> On success of ajax request, trigger another ajax call to a separate method which keep checking the status of the job after a fix interval say 3 seconds.
-> when the second ajax call see that status is complete, it will reload the page

So now, although user see some increase in time due to overhead introduced in checking status after every 3 seconds, Heroku see the request response cycle working as in every 3 second it see a request coming asking for the job status and a response going on with the current status : pending , queued, running etc .

So here is the code:

Add sidekiq to Gemfile and run bundle install

gem 'sidekiq'
gem 'sidekiq-status'

routes

  resources :loans
  post 'credit_request' => 'loans#check_credit_request', :as => :credit_request
  get 'check_job_status' => 'loans#check_job_status', :as => :loan_status

write the background code in lib folder say – lib/application.rb

require 'rest_client'
require 'base64'

module LoanPath
  class Application
    
    include Sidekiq::Worker
    include Sidekiq::Status::Worker
    sidekiq_options :retry => false
    
    def perform(*args)
      apply_credit(args[1])
      lp_status_code = background_task_output[:lpcode].present? ? background_task_output[:lpcode] : ""
      lp_status_message = background_task_output[:lp_message].present? ? background_task_output[:lp_message] : ""
      lp_data = background_task_output[:lp_data].present? ? background_task_output[:lp_data] : ""
      store :lp_status_code => lp_status_code
      store :lp_status_message => lp_status_message
      store :lp_data => lp_data
      at 100,100, background_task_output[:lp_message] 
                   if background_task_output[:lp_message].present?
    end
  end
  
  def apply_credit(detail)
    begin
      uri = APP_CONSTANTS["credit_request_endpoint"]
      payload = "whatever xml or other data you want to send"
       rest_resource = RestClient::Resource.new(uri, {:user => "your username", :password => "xyz", :timeout => 60, :open_timeout =>60})
      credit_submit = rest_resource.post payload, :content_type => "application/xml"
      {:lpcode => "LpValid", :lp_message => "Credit Request Submitted Successfully", :lp_data => credit_submit}
    rescue Exception => e
      error_message = "System encountered error, please try later"
      {:lpcode => "LpError", :lp_message => error_message, :lp_data => nil}
    end
  end
    
  def running_background_job(job_id)
    status = Sidekiq::Status::status(job_id)
    status_message = Sidekiq::Status::message(job_id)
    lp_status_code = Sidekiq::Status::get(job_id, :lp_status_code)
    lp_status_message = Sidekiq::Status::get(job_id, :lp_status_message)
    lp_data = Sidekiq::Status::get(job_id, :lp_data)
    {:status => status.to_s, :status_message => status_message, 
     :lp_status_code => lp_status_code, :lp_status_message => 
     lp_status_message,:lp_data => lp_data}
  end
  
end

The loan view to fill the details is as below:

<%= form_tag credit_request_path, :method => :post, :id => "credit-request" do %>
   your form fields
   <%= submit_tag 'Submit Credit Request', :id => 'apply-credit'%>
<% end %>

controller code which handle the loan submit is as below:

def credit_request
  job_id = LoanPath::Application.perform_async("apply_credit", params)
  render :json => {:job_id => job_id , :current_url => loan_url(params[:id])}
end

def check_job_status
  job_status = BackgroundJobs.new.running_background_job(params[:job_id])
  @application_saved = "Error"
  if (job_status[:lp_status_code].to_s == "LpValid")
    @application_saved = "Success"
    @message = "Credit Application Submitted Successfully."
  elsif (job_status[:lp_status_code].to_s == "LpError")
    @message = job_status[:lp_status_message].present? ? 
               job_status[:lp_status_message] : 
               "System encountered error, please try later"
  end
  render :json => {:status => job_status[:status], :redirect_to => params[:current_path], :message => @message, :application_status => @application_saved}
end

Now the most important, theĀ  js code which submit the form through ajax and keep checking status is as below

$('#apply-credit').click(function (e) {
    e.preventDefault();
    if ($('#accept_tc').is(':checked')) {
        var intervalId = '';
        
        $.ajax({
            url: "/loans/credit_request",
            data: $("#credit-request").serialize(),
            dataType: "json",
            type: "POST",
            success: function (job) {
              intervalId = setInterval(function () {
                  checkStatus(job,intervalId);
                }, 3000);
            }
          });
        
        $('#first_name').focus();
        $('#credit-request').block({
            message: null
          });
        return false;
      } else {
        alert('Please read and accept the terms, above, before submitting credit request.');
        $('#accept_tc').focus();
        return false;
      }
      
    })

function checkStatus(job, intervalId) {
    $.ajax({
        url: "/check_job_status",
        data: {
          job_id: job.job_id,
          current_path: job.current_url,
        },
        type: "get",
        dataType: "json",
        success: function (jobStatus) {
          console.log(jobStatus);
          if (jobStatus.status === 'complete' || jobStatus.status === "failed") {
              if (jobStatus.application_status === "Success") {
                  window.location = jobStatus.redirect_to;
                }
              else {
                  $('#ajax_message_success').hide();
                  $("#ajax_message_error").hide();
                  $("#ajax_message_error").text(jobStatus.message);
                  $('#ajax_message_error').show();
                  $("#application_tab").unblock();
                  $(window).scrollTop(0);
                }
                window.clearInterval(intervalId);
              }
            }
          });
      }

That’s all now Heroku will not complain about Timeout error