class Net::HTTP::Persistent
Persistent connections for Net::HTTP
Net::HTTP::Persistent maintains persistent connections across all the servers you wish to talk to. For each host:port you communicate with a single persistent connection is created.
Multiple Net::HTTP::Persistent objects will share the same set of connections.
For each thread you start a new connection will be created. A Net::HTTP::Persistent connection will not be shared across threads.
You can shut down the HTTP connections when done by calling shutdown. You should name your Net::HTTP::Persistent object if you intend to call this method.
Example:
require 'net/http/persistent' uri = URI 'http://example.com/awesome/web/service' http = Net::HTTP::Persistent.new 'my_app_name' # perform a GET response = http.request uri # create a POST post_uri = uri + 'create' post = Net::HTTP::Post.new post_uri.path post.set_form_data 'some' => 'cool data' # perform the POST, the URI is always required response http.request post_uri, post
SSL¶ ↑
SSL connections are automatically created depending upon the scheme of the URI. SSL connections are automatically verified against the default certificate store for your computer. You can override this by changing #verify_mode or by specifying an alternate cert_store.
Here are the SSL settings, see the individual methods for documentation:
- certificate
-
This client's certificate
- ca_file
-
The certificate-authority
- cert_store
-
An SSL certificate store
- private_key
-
The client's SSL private key
- reuse_ssl_sessions
-
Reuse a previously opened SSL session for a new connection
- ssl_version
-
Which specific SSL version to use
- verify_callback
-
For server certificate verification
- verify_mode
-
How connections should be verified
Proxies¶ ↑
A proxy can be set through proxy= or at initialization
time by providing a second argument to ::new. The proxy may be the URI of
the proxy server or :ENV
which will consult environment
variables.
See proxy= and proxy_from_env for details.
Headers¶ ↑
Headers may be specified for use in every request. headers are appended to any headers on the request. override_headers replace existing headers on the request.
The difference between the two can be seen in setting the User-Agent.
Using http.headers['User-Agent'] = 'MyUserAgent'
will
send “Ruby, MyUserAgent” while http.override_headers['User-Agent'] =
'MyUserAgent'
will send “MyUserAgent”.
Tuning¶ ↑
Segregation¶ ↑
By providing an application name to ::new you can separate your connections from the connections of other applications.
Idle Timeout¶ ↑
If a connection hasn't been used for 5 seconds it will automatically be reset upon the next use to avoid attempting to send to a closed connection. This can be adjusted through idle_timeout.
Reducing this value may help avoid the “too many connection resets” error when sending non-idempotent requests while increasing this value will cause fewer round-trips.
Read Timeout¶ ↑
The amount of time allowed between reading two chunks from the socket. Set through read_timeout
Open Timeout¶ ↑
The amount of time to wait for a connection to be opened. Set through open_timeout.
Idle Timeout¶ ↑
If a connection has not been used in this many seconds it will be reset when a request would use the connection. The default idle timeout is unlimited. If you know the server's idle timeout setting this value will eliminate failures from attempting non-idempotent requests on closed connections. Set through idle_timeout.
Socket Options¶ ↑
Socket options may be set on newly-created connections. See socket_options for details.
Non-Idempotent Requests¶ ↑
By default non-idempotent requests will not be retried per RFC 2616. By setting #retry_change_requests to true requests will automatically be retried once.
Only do this when you know that retrying a POST or other non-idempotent request is safe for your application and will not create duplicate resources.
The recommended way to handle non-idempotent requests is the following:
require 'net/http/persistent' uri = URI 'http://example.com/awesome/web/service' post_uri = uri + 'create' http = Net::HTTP::Persistent.new 'my_app_name' post = Net::HTTP::Post.new post_uri.path # ... fill in POST request begin response = http.request post_uri, post rescue Net::HTTP::Persistent::Error # POST failed, make a new request to verify the server did not process # the request exists_uri = uri + '...' response = http.get exists_uri # Retry if it failed retry if response.code == '404' end
The method of determining if the resource was created or not is unique to the particular service you are using. Of course, you will want to add protection from infinite looping.
Constants
- VERSION
The version of Net::HTTP::Persistent you are using
Attributes
An SSL certificate authority. Setting this will set #verify_mode to VERIFY_PEER.
An SSL certificate store. Setting this will override the default certificate store. See #verify_mode for more information.
This client's OpenSSL::X509::Certificate
Sends #debug_output to this IO via Net::HTTP#set_debug_output.
Never use this method in production code, it causes a serious security hole.
Headers that are added to every request using Net::HTTP#add_field
Maps host:port to an HTTP version. This allows us to enable version specific features.
Maximum time an unused connection can remain idle before being automatically closed.
The value sent in the Keep-Alive header. Defaults to 30. Not needed for HTTP/1.1 servers.
This may not work correctly for HTTP/1.0 servers
This method may be removed in a future version as RFC 2616 does not require this header.
A name for this connection. Allows you to keep your connections apart from everybody else's.
Seconds to wait until a connection is opened. See Net::HTTP#open_timeout
Headers that are added to every request using Net::HTTP#[]=
This client's SSL private key
The URL through which requests will be proxied
Seconds to wait until reading one block. See Net::HTTP#read_timeout
Enable retries of non-idempotent requests that change data (e.g. POST requests) when the server has disconnected.
This will in the worst case lead to multiple requests with the same data, but it may be useful for some applications. Take care when enabling this option to ensure it is safe to POST or perform other non-idempotent requests to the server.
By default SSL sessions are reused to avoid extra SSL handshakes. Set this to false if you have problems communicating with an HTTPS server like:
SSL_connect [...] read finished A: unexpected message (OpenSSL::SSL::SSLError)
An array of options for Socket#setsockopt.
By default the TCP_NODELAY option is set on sockets.
To set additional options append them to this array:
http.socket_options << [Socket::SOL_SOCKET, Socket::SO_KEEPALIVE, 1]
SSL version to use.
By default, the version will be negotiated automatically between client and server. Ruby 1.9 and newer only.
SSL verification callback. Used when #ca_file is set.
HTTPS verify mode. Defaults to OpenSSL::SSL::VERIFY_PEER which verifies the server certificate.
If no #ca_file or #cert_store is set the default system certificate store is used.
You can use verify_mode
to override any default values.
Public Class Methods
Creates a new Net::HTTP::Persistent.
Set name
to keep your connections apart from everybody
else's. Not required currently, but highly recommended. Your library
name should be good enough. This parameter will be required in a future
version.
proxy
may be set to a URI::HTTP or :ENV to pick up proxy
options from the environment. See #proxy_from_env for
details.
In order to use a URI for the proxy you may need to do some extra work beyond URI parsing if the proxy requires a password:
proxy = URI 'http://proxy.example' proxy.user = 'AzureDiamond' proxy.password = 'hunter2'
# File lib/net/http/persistent.rb, line 370 def initialize name = nil, proxy = nil @name = name @debug_output = nil @proxy_uri = nil @headers = {} @override_headers = {} @http_versions = {} @keep_alive = 30 @open_timeout = nil @read_timeout = nil @idle_timeout = 5 @socket_options = [] @socket_options << [Socket::IPPROTO_TCP, Socket::TCP_NODELAY, 1] if Socket.const_defined? :TCP_NODELAY key = ['net_http_persistent', name].compact @generation_key = [key, 'generations' ].join('_').intern @ssl_generation_key = [key, 'ssl_generations'].join('_').intern @request_key = [key, 'requests' ].join('_').intern @timeout_key = [key, 'timeouts' ].join('_').intern @certificate = nil @ca_file = nil @private_key = nil @ssl_version = nil @verify_callback = nil @verify_mode = OpenSSL::SSL::VERIFY_PEER @cert_store = nil @generation = 0 # incremented when proxy URI changes @ssl_generation = 0 # incremented when SSL session variables change @reuse_ssl_sessions = true @retry_change_requests = false self.proxy = proxy if proxy end
Public Instance Methods
Sets the SSL certificate authority file.
# File lib/net/http/persistent.rb, line 422 def ca_file= file @ca_file = file reconnect_ssl end
Is the request idempotent or is #retry_change_requests allowed
# File lib/net/http/persistent.rb, line 589 def can_retry? req retry_change_requests or idempotent?(req) end
Overrides the default SSL certificate store used for verifying connections.
# File lib/net/http/persistent.rb, line 432 def cert_store= store @cert_store = store reconnect_ssl end
Sets this client's OpenSSL::X509::Certificate
# File lib/net/http/persistent.rb, line 413 def certificate= certificate @certificate = certificate reconnect_ssl end
Workaround for missing Net::HTTPHeader#connection_close? on Ruby 1.8
# File lib/net/http/persistent.rb, line 597 def connection_close? header header.connection_close? end
Creates a new connection for uri
# File lib/net/http/persistent.rb, line 462 def connection_for uri Thread.current[@generation_key] ||= Hash.new { |h,k| h[k] = {} } Thread.current[@ssl_generation_key] ||= Hash.new { |h,k| h[k] = {} } Thread.current[@request_key] ||= Hash.new 0 Thread.current[@timeout_key] ||= Hash.new EPOCH use_ssl = uri.scheme.downcase == 'https' if use_ssl then ssl_generation = @ssl_generation ssl_cleanup ssl_generation connections = Thread.current[@ssl_generation_key][ssl_generation] else generation = @generation cleanup generation connections = Thread.current[@generation_key][generation] end net_http_args = [uri.host, uri.port] connection_id = net_http_args.join ':' if @proxy_uri then connection_id << @proxy_connection_id net_http_args.concat @proxy_args end connection = connections[connection_id] unless connection = connections[connection_id] then connections[connection_id] = http_class.new(*net_http_args) connection = connections[connection_id] ssl connection if use_ssl else last_used = Thread.current[@timeout_key][connection.object_id] reset connection unless last_used > max_age end unless connection.started? then connection.set_debug_output @debug_output if @debug_output connection.open_timeout = @open_timeout if @open_timeout connection.read_timeout = @read_timeout if @read_timeout connection.start socket = connection.instance_variable_get :@socket if socket then # for fakeweb @socket_options.each do |option| socket.io.setsockopt(*option) end end end connection rescue Errno::ECONNREFUSED raise Error, "connection refused: #{connection.address}:#{connection.port}" rescue Errno::EHOSTDOWN raise Error, "host down: #{connection.address}:#{connection.port}" end
Workaround for missing Net::HTTPHeader#connection_keep_alive? on Ruby 1.8
# File lib/net/http/persistent.rb, line 604 def connection_keep_alive? header header.connection_keep_alive? end
Returns an error message containing the number of requests performed on this connection
# File lib/net/http/persistent.rb, line 530 def error_message connection requests = Thread.current[@request_key][connection.object_id] - 1 # fixup last_use = Thread.current[@timeout_key][connection.object_id] age = Time.now - last_use "after #{requests} requests on #{connection.object_id}, " "last used #{age} seconds ago" end
URI::escape wrapper
# File lib/net/http/persistent.rb, line 543 def escape str CGI.escape str if str end
Finishes the Net::HTTP connection
# File lib/net/http/persistent.rb, line 550 def finish connection, thread = Thread.current if requests = thread[@request_key] then requests.delete connection.object_id end connection.finish rescue IOError end
Returns the HTTP protocol version for
uri
# File lib/net/http/persistent.rb, line 571 def http_version uri @http_versions["#{uri.host}:#{uri.port}"] end
Is req
idempotent according to RFC 2616?
# File lib/net/http/persistent.rb, line 578 def idempotent? req case req when Net::HTTP::Delete, Net::HTTP::Get, Net::HTTP::Head, Net::HTTP::Options, Net::HTTP::Put, Net::HTTP::Trace then true end end
If a connection hasn't been used since #max_age it will be reset and reused
# File lib/net/http/persistent.rb, line 628 def max_age Time.now - @idle_timeout end
Adds “http://” to the String uri
if it is missing.
# File lib/net/http/persistent.rb, line 635 def normalize_uri uri (uri =~ /^https?:/) ? uri : "http://#{uri}" end
Pipelines requests
to the HTTP
server at uri
yielding responses if a block is given. Returns
all responses recieved.
See Net::HTTP::Pipeline for further details.
Only if net-http-pipeline
was required before
net-http-persistent
pipeline will be present.
# File lib/net/http/persistent.rb, line 650 def pipeline uri, requests, &block # :yields: responses connection = connection_for uri connection.pipeline requests, &block end
Sets this client's SSL private key
# File lib/net/http/persistent.rb, line 659 def private_key= key @private_key = key reconnect_ssl end
Sets the proxy server. The proxy
may be the URI of the proxy
server, the symbol :ENV
which will read the proxy from the
environment or nil to disable use of a proxy. See proxy_from_env for
details on setting the proxy from the environment.
If the proxy URI is set after requests have been made, the next request will shut-down and re-open all connections.
If you are making some requests through a proxy and others without a proxy use separate Net::Http::Persistent instances.
# File lib/net/http/persistent.rb, line 677 def proxy= proxy @proxy_uri = case proxy when :ENV then proxy_from_env when URI::HTTP then proxy when nil then # ignore else raise ArgumentError, 'proxy must be :ENV or a URI::HTTP' end if @proxy_uri then @proxy_args = [ @proxy_uri.host, @proxy_uri.port, @proxy_uri.user, @proxy_uri.password, ] @proxy_connection_id = [nil, *@proxy_args].join ':' end reconnect reconnect_ssl end
Creates a URI for an HTTP proxy server from ENV variables.
If HTTP_PROXY
is set a proxy will be returned.
If HTTP_PROXY_USER
or HTTP_PROXY_PASS
are set the
URI is given the indicated user and password unless HTTP_PROXY contains
either of these in the URI.
For Windows users, lowercase ENV variables are preferred over uppercase ENV variables.
# File lib/net/http/persistent.rb, line 712 def proxy_from_env env_proxy = ENV['http_proxy'] || ENV['HTTP_PROXY'] return nil if env_proxy.nil? or env_proxy.empty? uri = URI normalize_uri env_proxy unless uri.user or uri.password then uri.user = escape ENV['http_proxy_user'] || ENV['HTTP_PROXY_USER'] uri.password = escape ENV['http_proxy_pass'] || ENV['HTTP_PROXY_PASS'] end uri end
Forces reconnection of HTTP connections.
# File lib/net/http/persistent.rb, line 730 def reconnect @generation += 1 end
Forces reconnection of SSL connections.
# File lib/net/http/persistent.rb, line 737 def reconnect_ssl @ssl_generation += 1 end
Makes a request on uri
. If req
is nil a
Net::HTTP::Get is performed against uri
.
If a block is passed request behaves like Net::HTTP#request (the body of the response will not have been read).
req
must be a Net::HTTPRequest subclass (see Net::HTTP for a list).
If there is an error and the request is idempontent according to RFC 2616 it will be retried automatically.
# File lib/net/http/persistent.rb, line 769 def request uri, req = nil, &block retried = false bad_response = false req = Net::HTTP::Get.new uri.request_uri unless req @headers.each do |pair| req.add_field(*pair) end @override_headers.each do |name, value| req[name] = value end unless req['Connection'] then req.add_field 'Connection', 'keep-alive' req.add_field 'Keep-Alive', @keep_alive end connection = connection_for uri connection_id = connection.object_id begin Thread.current[@request_key][connection_id] += 1 response = connection.request req, &block if connection_close?(req) or (response.http_version <= '1.0' and not connection_keep_alive?(response)) or connection_close?(response) then connection.finish end rescue Net::HTTPBadResponse => e message = error_message connection finish connection raise Error, "too many bad responses #{message}" if bad_response or not can_retry? req bad_response = true retry rescue IOError, EOFError, Timeout::Error, Errno::ECONNABORTED, Errno::ECONNRESET, Errno::EPIPE, Errno::EINVAL, OpenSSL::SSL::SSLError => e if retried or not can_retry? req due_to = "(due to #{e.message} - #{e.class})" message = error_message connection finish connection raise Error, "too many connection resets #{due_to} #{message}" end reset connection retried = true retry ensure Thread.current[@timeout_key][connection_id] = Time.now end @http_versions["#{uri.host}:#{uri.port}"] ||= response.http_version response end
Finishes then restarts the Net::HTTP
connection
# File lib/net/http/persistent.rb, line 744 def reset connection Thread.current[@request_key].delete connection.object_id Thread.current[@timeout_key].delete connection.object_id finish connection connection.start rescue Errno::ECONNREFUSED raise Error, "connection refused: #{connection.address}:#{connection.port}" rescue Errno::EHOSTDOWN raise Error, "host down: #{connection.address}:#{connection.port}" end
Shuts down all connections for thread
.
Uses the current thread by default.
If you've used Net::HTTP::Persistent across multiple threads you should call this in each thread when you're done making HTTP requests.
NOTE: Calling shutdown for another thread can be dangerous!
If the thread is still using the connection it may cause an error! It is best to call shutdown in the thread at the appropriate time instead!
# File lib/net/http/persistent.rb, line 850 def shutdown thread = Thread.current generation = reconnect cleanup generation, thread, @generation_key ssl_generation = reconnect_ssl cleanup ssl_generation, thread, @ssl_generation_key thread[@request_key] = nil thread[@timeout_key] = nil end
Shuts down all connections in all threads
NOTE: THIS METHOD IS VERY DANGEROUS!
Do not call this method if other threads are still using their connections! Call shutdown at the appropriate time instead!
Use this method only as a last resort!
# File lib/net/http/persistent.rb, line 871 def shutdown_in_all_threads Thread.list.each do |thread| shutdown thread end nil end
Enables SSL on connection
# File lib/net/http/persistent.rb, line 882 def ssl connection connection.use_ssl = true connection.ssl_version = @ssl_version if @ssl_version connection.verify_mode = @verify_mode if OpenSSL::SSL::VERIFY_PEER == OpenSSL::SSL::VERIFY_NONE and not Object.const_defined?(:I_KNOW_THAT_OPENSSL_VERIFY_PEER_EQUALS_VERIFY_NONE_IS_WRONG) then warn <<-WARNING !!!SECURITY WARNING!!! The SSL HTTP connection to: #{connection.address}:#{connection.port} !!!MAY NOT BE VERIFIED!!! On your platform your OpenSSL implementation is broken. There is no difference between the values of VERIFY_NONE and VERIFY_PEER. This means that attempting to verify the security of SSL connections may not work. This exposes you to man-in-the-middle exploits, snooping on the contents of your connection and other dangers to the security of your data. To disable this warning define the following constant at top-level in your application: I_KNOW_THAT_OPENSSL_VERIFY_PEER_EQUALS_VERIFY_NONE_IS_WRONG = nil WARNING end if @ca_file then connection.ca_file = @ca_file connection.verify_mode = OpenSSL::SSL::VERIFY_PEER connection.verify_callback = @verify_callback if @verify_callback end if @certificate and @private_key then connection.cert = @certificate connection.key = @private_key end connection.cert_store = if @cert_store then @cert_store else store = OpenSSL::X509::Store.new store.set_default_paths store end end
SSL version to use
# File lib/net/http/persistent.rb, line 947 def ssl_version= ssl_version @ssl_version = ssl_version reconnect_ssl end
SSL verification callback.
# File lib/net/http/persistent.rb, line 969 def verify_callback= callback @verify_callback = callback reconnect_ssl end
Sets the HTTPS verify mode. Defaults to OpenSSL::SSL::VERIFY_PEER.
Setting this to VERIFY_NONE is a VERY BAD IDEA and should NEVER be used. Securely transfer the correct certificate and update the default certificate store or set the ca file instead.
# File lib/net/http/persistent.rb, line 960 def verify_mode= verify_mode @verify_mode = verify_mode reconnect_ssl end