Archive for the ‘study notes’ Category
Javascript: Sync JSON Data
/** * Synchronize new data to original object and keep reference * Check object data type and traverse object tree to delete old key * as well as replace with new data: * * - if keyed data is directory (hash map) 'object', sync recursively * - else (including number, string, and array, etc), replace with new data * * @returns void. */ function syncObjectData(newData, orgData) { if (newData != null && orgData !== newData) { var sourceType = orgData.toType(); var targetType = newData.toType(); if (sourceType != targetType) { // the top level type must match var message = "Cannot update '" + sourceType + "' with different type " + targetType; throw new Error(message); } else if (sourceType != 'object') { // can only sync object type throw new Error("Does not support updating type of " + orgType); } else { for (var key in newData) { if (newData.hasOwnProperty(key)) { if (orgData.hasOwnProperty(key) && newData[key] != null) { var orgType = toType(orgData[key]); var newType = toType(newData[key]); if (newType == orgType && orgType == 'object' && orgData[key] != null) { syncObjectData(newData[key], orgData[key]); } else { orgData[key] = newData[key]; } } else if (!orgData.hasOwnProperty(key)) { orgData[key] = newData[key]; } else { delete orgData[key]; } } } for (var key in orgData) { if (orgData.hasOwnProperty(key)) { if (!newData.hasOwnProperty(key) || newData[key] == null) { delete orgData[key]; } } } } } }
Here is the helper function to get name (string) of the exact type.
/** * Convert data type to string, e.g. 'undefined', * or 'null', 'boolean', 'number', 'string', 'array', 'object', 'date', 'function', etc. * https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects */ function toType(variable) { return ({}).toString.call(variable).match(/\s([a-zA-Z]+)/)[1].toLowerCase(); }
Python class vs module
class
mechanism to accomplish such task. Let’s take a first look on python class.
1. Python class mechanism
The following demo is to write a swiftclient wrapper to access object store on OpenStack (cloud computing platform). Since swift client API methods requires certain dependencies (e.g. auth token, swift URL, and container name), it would be nice to wrap them in a class object instead of calling API methods with these in parameters. For example, ideal coding in application domain should look like this –
from keystone import get_auth_token from swift import get_swift, Swift # Use built-in swift_class module to get an instance of Swift swift = get_swift() # Otherwise, if dependencies (e.g. swift_url and container_name) in app domain auth_token = get_auth_token() swift = Swift(auth_token, swift_url, container_name) # Once we have a Swift instance ... afile = swift.get_file_contents('foo.txt') files = swift.get_files_in_container()
Notice that if application settings need to be separated from swift module (most likely in a large-scale project), the class instantiation will be in application domain (where e.g. settings belong to); otherwise, the swift module could have a get_swift()
function to construct a swift instance (like in this demo). The class module and unit tests are listed as below.
See common/swift.py
(Swift
class) –
# swift.py (Swift class) import logging import mimetypes import sys from swiftclient import client as swift_client from swiftclient.exceptions import ClientException from keystone import get_auth_token from config import settings logger = logging.getLogger(__name__) # ================================================ # Swift instance initialization # ================================================ def get_swift(): """ Get a Swift instance per application settings """ auth_token = get_auth_token() container = settings('swift_container') swift_url = settings('swift_url') swift = Swift(auth_token, swift_url, container) return swift # ================================================ # Swift class # ================================================ class Swift(object): def __init__(self, auth_token, swift_url, container_name): """ Initialize a Swift instance """ self.auth_token = auth_token self.swift_url = swift_url self.container = container_name self.connection = self._get_connection() def _get_connection(self): try: return swift_client.Connection( preauthurl=self.swift_url, preauthtoken=self.auth_token, retries=5, auth_version='1', insecure=True) except Exception as e: err_message = "Exception raised initiating a swift connection." logger.exception(err_message) raise def check_container(self): """ Determine if default container missing in Swift """ try: headers, container_list = self.connection.get_account() for container in container_list: if container['name'] == self.container: return True return False except Exception as e: err_message = "Exception raised on checking container exists." logger.exception(err_message) raise def ensure_container_exists(self): """ Ensure default container exists in Swift. """ # Determine if necessary container missing; if so, create it container_exists = self.check_container() if (not container_exists): try: response = {} self.connection.put_container( self.container, response_dict=response) except Exception as e: err = "Exception on creating container {0}.".format( self.container) logger.exception(err) raise def get_file_contents(self, file_name): """ Function wrapper to perform 'get_object' call on Swift """ try: response_dict = {} # Response from Swift: # a tuple of (response headers, the object contents) # The response headers will be a dict and all header names # will be lowercase. response = self.connection.get_object( self.container, file_name, response_dict=response_dict) file_contents = response[1] return file_contents except Exception as e: err = "Exception on getting {0} from Swift.".format(file_name) logger.exception(err) raise def get_files_in_container(self): result = self.connection.get_container( self.container, full_listing=True) return result[1] def save_file(self, file_name, file_contents): """ Function wrapper to perform 'put_object' call Swift """ try: self.ensure_container_exists() response = {} # Example of response from put_object call - # { # 'status': 201, # 'headers': { # 'content-length': '0', # 'last-modified': 'Fri, 17 Jul 2015 04:43:56 GMT', # 'connection': 'keep-alive', # 'etag': 'd41d8cd98f00b204e9800998ecf8427e', # 'x-trans-id': 'txeddbca07d8e744deae343-0055a8880c', # 'date': 'Fri, 17 Jul 2015 04:43:57 GMT', # 'content-type': 'text/html; charset=UTF-8'}, # 'reason': 'Created', # 'response_dicts': [{ # 'status': 201, # 'headers': { # 'content-length': '0', # 'last-modified': # 'Fri, 17 Jul 2015 04:43:56 GMT', # 'connection': 'keep-alive', # 'etag': 'd41d8cd98f00b204e9800998ecf8427e', # 'x-trans-id': 'txeddbca07d8e744deae343-0055a8880c', # 'date': 'Fri, 17 Jul 2015 04:43:57 GMT', # 'content-type': 'text/html; charset=UTF-8'}, # 'reason': 'Created'}]} self.connection.put_object( self.container, file_name, file_contents, content_length=sys.getsizeof(file_contents), content_type=mimetypes.guess_type(file_name, strict=True)[0], response_dict=response) return response except Exception as e: err_message = "Exception on saving file contents to Swift.\n" logger.exception(err_message) raise
See common/tests/swift_tests.py
(Unit tests for Swift
class) –
# swift_tests.py import logging import mock import os import StringIO import sys import tarfile import unittest from pyramid import testing from mock import Mock, MagicMock, patch, mock_open from swiftclient.exceptions import ClientException from swift import Swift logger = logging.getLogger(__name__) class SwiftClassTests(unittest.TestCase): @patch('swiftclient.client.Connection') def setUp(self, mock_connection): self.config = testing.setUp() self.auth_token = Mock() self.account_data = ([{}], [{'name': 'container1'}]) self.container = 'container1' self.container_data = ([{}], [{'name': 'file1'}, {'name': 'file2'}]) self.swift_url = 'http://0.0.0.0' self.connection = Mock() # patch('common.swift.Swift.get_connection', # return_value=self.connection mock_connection.return_value = self.connection self.swift = Swift( self.auth_token, self.swift_url, self.container) def tearDown(self): testing.tearDown() @patch('swiftclient.client.Connection') def test_constructor(self, mock_connection): self.assertEqual(self.swift.auth_token, self.auth_token) self.assertEqual(self.swift.swift_url, self.swift_url) self.assertEqual(self.swift.connection, self.connection) self.assertEqual(self.swift.container, self.container) @patch('swiftclient.client.Connection') def test_connection_exception(self, mock_connection): error_message = 'CONNECTION ERROR' mock_connection.side_effect = Exception(error_message) with self.assertRaises(Exception) as cm: result = Swift(self.auth_token, self.swift_url, self.container) self.assertEqual(str(cm.exception), error_message) def test_check_container_exception(self): name = 'foo' error_message = 'PUT CONTAINER ERROR' self.swift.connection.get_account.side_effect \ = Exception(error_message) with self.assertRaises(Exception) as cm: result = self.swift.check_container() self.assertEqual(str(cm.exception), error_message) self.assertFalse(result) def test_check_container_when_false(self): self.swift.connection.get_account.return_value = self.account_data self.swift.container = 'dummy' result = self.swift.check_container() self.assertFalse(result) def test_check_container_when_true(self): self.swift.connection.get_account.return_value = self.account_data result = self.swift.check_container() self.assertTrue(result) def test_check_file_exists(self): self.swift.check_container = MagicMock() self.swift.check_container.return_value = True self.swift.connection.get_container = MagicMock() self.swift.connection.get_container.return_value = self.container_data result = self.swift.check_file_exists('fileX') self.assertFalse(result) result = self.swift.check_file_exists('file1') self.assertTrue(result) def test_check_file_exists_no_container(self): self.swift.check_container = MagicMock() self.swift.check_container.return_value = False result = self.swift.check_file_exists('filename') self.assertFalse(result) def test_ensure_container_exists(self): success = {'status': 200} def mock_put_success(container_name, response_dict): logger.debug( 'Called with {0} {1}'.format(container_name, response_dict)) response_dict['status'] = success['status'] with patch.object( self.swift.connection, 'get_account', return_value=self.account_data): with patch.object( self.swift.connection, 'put_container', side_effect=mock_put_success) as mocked_put: self.swift.ensure_container_exists() mocked_put.assert_called() def test_ensure_container_exists_exception(self): error_message = 'PUT CONTAINER ERROR' with patch.object( self.swift.connection, 'get_account', return_value=self.account_data): with patch.object( self.swift.connection, 'put_container', side_effect=Exception(error_message)) as mocked_put: self.swift.check_container = MagicMock() self.swift.check_container.return_value = False with self.assertRaises(Exception) as cm: self.swift.ensure_container_exists() self.assertEqual(str(cm.exception), error_message) def test_get_file_contents(self): response = ([{}], '_filecontents') self.swift.connection.get_object = MagicMock() self.swift.connection.get_object.return_value = response result = self.swift.get_file_contents('file_name') self.assertEqual(result, response[1]) def test_get_file_contents_exeption(self): error_message = 'Exception on get object' self.swift.connection.get_object = MagicMock() self.swift.connection.get_object.side_effect = Exception(error_message) with self.assertRaises(Exception) as cm: result = self.swift.get_file_contents('file_name') self.assertEqual(str(cm.exception), error_message) def test_get_files_in_container(self): self.swift.connection.get_container = MagicMock() self.swift.connection.get_container.return_value = self.container_data result = self.swift.get_files_in_container() self.assertEqual(result, self.container_data[1]) @patch('mimetypes.guess_type') @patch('sys.getsizeof') def test_save_file_contents(self, mock_getsizeof, mock_guess_type): success = {'status': 200} def mock_put_success( container_name, file_name, contents, content_length, content_type, response_dict): response_dict['status'] = success['status'] file_name = 'filename' contents = MagicMock() mock_getsizeof.return_value = 999 mock_guess_type.return_value = ['filetype'] with mock.patch.object( self.swift.connection, 'put_object', side_effect=mock_put_success) as mocked_put: self.swift.check_container = MagicMock() self.swift.check_container.return_value = True self.swift.save_file_contents(file_name, contents) mocked_put.assert_called_with( self.container, file_name, contents, content_length=999, content_type='filetype', response_dict=success) @patch('mimetypes.guess_type') @patch('sys.getsizeof') def test_save_file_contents_Exception( self, mock_getsizeof, mock_guess_type): file_name = 'filename' contents = MagicMock() mock_getsizeof.return_value = 999 mock_guess_type.return_value = ['filetype'] error_message = "SWIFT PUT OBJECT ERROR" self.swift.connection.put_object.side_effect = Exception(error_message) self.swift.check_container = MagicMock() self.swift.check_container.return_value = False with self.assertRaises(Exception) as cm: self.swift.save_file_contents(file_name, contents) self.assertEqual(str(cm.exception), error_message)
2. Python module
In use of Python class mechanism (as in above example), it takes a couple steps (of initialization, if no dependency injection used) before all class methods are accessible. Next, let’s see a comparison on how to make swift
functions (more in a traditional python style) available right after import
the module, in order to have fewer lines (under certain conditions, see discussion on next) of code as demonstrated below. The full source code (with config
, keystone
modules, and unit tests) are included at the end.
Beware of how @checks_config
decorator is used, in swift_module.py
, to guarantee properly instantiating a SwiftConfig
if any swift
method is called without one. One benefit using optional config=None
parameter is to have some flexibility that a config
can be either specified by the caller or created by swift
module automatically. This makes swift
module methods have same signatures (without config
), comparing to class mechanism. But disadvantage (of such hiding) also appears that each auto-creation of a config
by the decorator will have a different instance.
import swift_module as swift # optionally to get a swift config first (see SwiftConfig in swift_module.py) config = swift.get_swift_config() # swift module functions are available after the import a_file = swift.get_file_contents('foo.txt', config=config) allist = swift.get_files_in_container(config=config)
import logging import pyramid logger = logging.getLogger(__name__) def checks_config(config_func): """ Get decorator to use config_func as initiator for 'config' arg Keyword arguments: config_func -- a function to get proper configuration """ def checks_config_decorator(original_func): """ Call decorated original_func with checking its 'config' arg Keyword arguments: func -- original function to be decorated """ def _arg_index_of(func, name): """ Get the index of a named arg on a func call """ import inspect argspec = inspect.getargspec(func) for i in range(len(argspec[0])): if (argspec[0][i] == name): logger.debug("argspec[0][{0}]=={1}".format(i, name)) return i return -1 def _checks_config_wrapper(*args, **kwargs): """ Check 'config' arg before calling original_func Call specified config_func if 'config' arg value is None. """ arg_idx = _arg_index_of(original_func, 'config') has_arg = 0 <= arg_idx and arg_idx < len(args) arg_cfg = args[arg_idx] if (has_arg) else None kwa_cfg = kwargs.get('config') if (kwa_cfg is None and arg_cfg is None): # logger.debug('Getting config by {0}'.format(config_func)) kwargs['config'] = config_func() return original_func(*args, **kwargs) # calls the original function with checking proper configuration return _checks_config_wrapper # returns a decorated function return checks_config_decorator def settings(item): """ Get a reference to the settings in the .ini file """ registry = pyramid.threadlocal.get_current_registry() return registry.settings.get(item, None)
from config import settings from keystoneclient.v2_0 import client as keystone_client from logging import getLogger logger = getLogger(__name__) def get_auth_token(): """ Get an auth token from Keystone. """ try: keystone = keystone_client.Client( username=settings('cloud_username'), password=settings('cloud_password'), tenant_name=settings('cloud_project_name'), auth_url=settings('cloud_auth_url'), insecure=True) return keystone.auth_ref['token']['id'] except Exception as e: logger.error( "Exception authenticating against Keystone") logger.exception("Details: {0}".format(e)) raise
# swift_module.py import logging import mimetypes import sys from config import checks_config, settings from keystone import get_auth_token from swiftclient import client as swift_client from swiftclient.exceptions import ClientException logger = logging.getLogger(__name__) # ================================================ # Swift configuration class # ================================================ class SwiftConfig(object): def __init__(self, auth_token, swift_url, container_name): """ Initialize a Swift configuration instance """ self.auth_token = auth_token self.swift_url = swift_url self.container = container_name self.connection = self._get_connection() def _get_connection(self): """ Get a connection to Swift object store """ try: return swift_client.Connection( preauthurl=self.swift_url, preauthtoken=self.auth_token, retries=5, auth_version='1', insecure=True) except Exception as e: err_message = "Exception raised initiating a swift connection." logger.exception(err_message) raise # ToDo [zhuyux]: considering singleton for SwiftConfig instance _swift_config_singleton = None # ================================================ # Swift configuration initialization # ================================================ def get_swift_config(): """ Get a SwiftConfig instance per application settings """ auth_token = get_auth_token() container = settings('swift_container') swift_url = settings('swift_url') swift_cfg = SwiftConfig(auth_token, swift_url, container) return swift_cfg def _get_config(): """ This is a fixed/non-mockable func pointer for @checks_config decorator """ # logger.debug('get_swift_config={0}'.format(get_swift_config)) return get_swift_config() # ================================================ # Swift module interfaces # ================================================ @checks_config(config_func=_get_config) def check_container(config=None): """ Check if default container exists in Swift Keyword arguments: config -- an instance of SwiftConfig (optional, default None) """ try: logger.debug('Checking container {0}'.format(config.container)) headers, container_list = config.connection.get_account() for container in container_list: logger.debug("--- container: {0}".format(container['name'])) if (container['name'] == config.container): logger.debug('--- found {0}'.format(config.container)) return False logger.debug('--- missing container {0}'.format(config.container)) return True except Exception as e: err_message = "Exception raised on checking container exists." logger.exception(err_message) raise @checks_config(config_func=_get_config) def check_file_exists(file_name, config=None): """ Check if specified file exists in Swift store Keyword arguments: file_name -- the name of the file to be checked in Swift store config -- an instance of SwiftConfig (optional, default None) """ if (check_container(config=config)): files = get_files_in_container(config=config) for file in files: if (file['name'] == file_name): return True return False @checks_config(config_func=_get_config) def ensure_container_exists(config=None): """ Ensure default container exists in Swift. Keyword arguments: config -- an instance of SwiftConfig (optional, default None) """ container_exists = check_container(config=config) if (not container_exists): try: response = {} config.connection.put_container( config.container, response_dict=response) logger.debug( "--- Container {0} created".format(config.container)) logger.debug("--- Response {0}".format(response)) except Exception as e: err = "Exception on creating container {0}.".format( config.container) logger.exception(err) raise @checks_config(config_func=_get_config) def get_file_contents(file_name, config=None): """ Function wrapper to perform 'get_object' call on Swift Keyword arguments: file_name -- the name of the file in Swift store config -- an instance of SwiftConfig (optional, default None) """ try: response_dict = {} # Response from Swift: # a tuple of (response headers, the object contents) # The response headers will be a dict and all header names # will be in lower case. response = config.connection.get_object( config.container, file_name, response_dict=response_dict) file_contents = response[1] return file_contents except Exception as e: err = "Exception on getting {0} from Swift.".format(file_name) logger.exception(err) raise @checks_config(config_func=_get_config) def get_files_in_container(config=None): """ Get info of all files in default Swift container Keyword arguments: config -- an instance of SwiftConfig (optional, default None) """ result = config.connection.get_container( config.container, full_listing=True) return result[1] @checks_config(config_func=_get_config) def save_file(file_name, file_contents, config=None): """ Function wrapper to perform 'put_object' call Swift Keyword arguments: file_name -- the name of the file to be saved file_contents -- the contents of the file to be saved in Swift store config -- an instance of SwiftConfig (optional, default None) """ try: # Ensure the default container exists ensure_container_exists(config=config) # Push the file contents to Swift response = {} config.connection.put_object( config.container, file_name, file_contents, content_length=sys.getsizeof(file_contents), content_type=mimetypes.guess_type(file_name, strict=True)[0], response_dict=response) return response except Exception as e: err = "Exception on saving file contents to Swift.\n" logger.exception(err) raise
See common/tests/swift_module_tests.py
import logging import mock import os import StringIO import sys import tarfile import unittest import common.swift_module as swift from pyramid import testing from mock import Mock, MagicMock, patch, mock_open from swiftclient.exceptions import ClientException logger = logging.getLogger(__name__) class SwiftTests(unittest.TestCase): @patch('common.swift_module.get_auth_token') @patch('swiftclient.client.Connection') def setUp(self, mock_swift_connection, mock_get_auth_token): self.config = testing.setUp() self.auth_token = MagicMock() self.account_data = ([{}], [{'name': 'container1'}]) self.container = 'container1' self.container_data = ([{}], [{'name': 'file1'}, {'name': 'file2'}]) self.swift_url = 'http://0.0.0.0' self.setting = 'dummy setting' self.connection = Mock() mock_swift_connection.return_value = self.connection mock_get_auth_token.return_value = self.auth_token self.swift_cfg = swift.SwiftConfig( self.auth_token, self.swift_url, self.container) self.swift_cfg.connection.get_account.return_value = self.account_data def tearDown(self): testing.tearDown() @patch('swiftclient.client.Connection') def test_swift_config(self, mock_connection): self.assertEqual(self.swift_cfg.auth_token, self.auth_token) self.assertEqual(self.swift_cfg.swift_url, self.swift_url) self.assertEqual(self.swift_cfg.connection, self.connection) self.assertEqual(self.swift_cfg.container, self.container) @patch('swiftclient.client.Connection') def test_swift_config_exception(self, mock_connection): error_message = 'CONNECTION ERROR' mock_connection.side_effect = Exception(error_message) with self.assertRaises(Exception) as cm: result = swift.SwiftConfig( self.auth_token, self.swift_url, self.container) self.assertEqual(str(cm.exception), error_message) def test_check_container_exception(self): error_message = 'GET ACCOUNT ERROR' self.swift_cfg.connection.get_account.side_effect \ = Exception(error_message) with self.assertRaises(Exception) as cm: result = swift.check_container(config=self.swift_cfg) self.assertEqual(str(cm.exception), error_message) self.assertFalse(result) @patch('common.swift_module.get_swift_config') def test_check_container_when_false(self, mock_get_config): mock_get_config.return_value = self.swift_cfg self.swift_cfg.container = '-=#=-dummy-=#=-' result = swift.check_container() self.assertFalse(result) @patch('common.swift_module.get_auth_token') @patch('swiftclient.client.Connection') def test_check_container_when_true( self, mock_swift_connection, mock_get_auth_token): mock_swift_connection.return_value = self.connection mock_get_auth_token.return_value = self.auth_token result = swift.check_container(config=self.swift_cfg) self.assertTrue(result) self.swift_cfg.container = '-=#=-dummy-=#=-' result = swift.check_container(config=self.swift_cfg) self.assertFalse(result) @patch('common.swift_module.get_swift_config') @patch('common.swift_module.check_container') def test_check_file_exists(self, mock_check_container, mock_get_config): mock_check_container.return_value = True mock_get_config.return_value = self.swift_cfg self.swift_cfg.connection.get_container = MagicMock() self.swift_cfg.connection.get_container.return_value = \ self.container_data result = swift.check_file_exists('fileX') self.assertFalse(result) result = swift.check_file_exists('file1') self.assertTrue(result) @patch('common.swift_module.check_container') def test_check_file_exists_no_container(self, mock_check_container): mock_check_container.return_value = False result = swift.check_file_exists('filename', config=self.swift_cfg) self.assertFalse(result) def test_ensure_container_exists(self): success = {'status': 200} def mock_put_success(container_name, response_dict): logger.debug( 'Called with {0} {1}'.format(container_name, response_dict)) response_dict['status'] = success['status'] with patch.object( self.swift_cfg.connection, 'get_account', return_value=self.account_data): with patch.object( self.swift_cfg.connection, 'put_container', side_effect=mock_put_success) as mocked_put: swift.ensure_container_exists(config=self.swift_cfg) mocked_put.assert_called() def test_ensure_container_exists_exception(self): error_message = 'PUT CONTAINER ERROR' with patch.object( self.swift_cfg.connection, 'get_account', return_value=self.account_data): with patch.object( self.swift_cfg.connection, 'put_container', side_effect=Exception(error_message)) as mocked_put: with self.assertRaises(Exception) as cm: import common.swift swift.ensure_container_exists(config=self.swift_cfg) self.assertEqual(str(cm.exception), error_message) def test_get_file_contents(self): import common.swift response = ([{}], '_filecontents') self.swift_cfg.connection.get_object = MagicMock() self.swift_cfg.connection.get_object.return_value = response result = swift.get_file_contents('file_name', config=self.swift_cfg) self.assertEqual(result, response[1]) def test_get_file_contents_exeption(self): error_message = 'Exception on get object' self.swift_cfg.connection.get_object = MagicMock() self.swift_cfg.connection.get_object.side_effect = Exception(error_message) with self.assertRaises(Exception) as cm: swift.get_file_contents('file_name', config=self.swift_cfg) self.assertEqual(str(cm.exception), error_message) def test_get_files_in_container(self): self.swift_cfg.connection.get_container = MagicMock() self.swift_cfg.connection.get_container.return_value = \ self.container_data result = swift.get_files_in_container(config=self.swift_cfg) self.assertEqual(result, self.container_data[1]) @patch('common.config.settings') @patch('common.swift_module.get_auth_token') @patch('swiftclient.client.Connection') def test_get_swift_config( self, mock_connection, mock_get_auth_token, mock_settings): mock_connection.return_value = self.connection mock_get_auth_token.return_value = self.auth_token mock_settings.return_value = self.setting result = swift.get_swift_config() self.assertEqual(result.auth_token, self.auth_token) self.assertEqual(result.connection, self.connection) self.assertEqual(result.container, self.setting) self.assertEqual(result.swift_url, self.setting) @patch('mimetypes.guess_type') @patch('sys.getsizeof') def test_save_file(self, mock_getsizeof, mock_guess_type): success = {'status': 200} def mock_put_success( container_name, file_name, contents, content_length, content_type, response_dict): response_dict['status'] = success['status'] mock_getsizeof.return_value = 999 mock_guess_type.return_value = ['filetype'] with mock.patch.object( self.swift_cfg.connection, 'put_object', side_effect=mock_put_success) as mocked_put: import common.swift swift.check_container = MagicMock() swift.check_container.return_value = True filename = 'filename' contents = MagicMock() swift.save_file(filename, contents, config=self.swift_cfg) mocked_put.assert_called_with( self.container, filename, contents, content_length=999, content_type='filetype', response_dict=success) @patch('mimetypes.guess_type') @patch('sys.getsizeof') def test_save_file_Exception( self, mock_getsizeof, mock_guess_type): import common.swift mock_getsizeof.return_value = 999 mock_guess_type.return_value = ['filetype'] error_message = "SWIFT PUT OBJECT ERROR" self.swift_cfg.connection.put_object.side_effect = \ Exception(error_message) swift.check_container = MagicMock() swift.check_container.return_value = False with self.assertRaises(Exception) as cm: filename = 'filename' contents = MagicMock() swift.save_file(filename, contents, config=self.swift_cfg) self.assertEqual(str(cm.exception), error_message)
Summary
Programmers like to write simple code (to make it clear and easy to understand, thus easier to share, test, and maintain). Sure there is always a cost of effort. In OO design (either with C++, Java, or C#), dependency injection pattern has been applied, not only to write less and cleaner code, but also to support the concept of “programming to interfaces, not implementations” – forcing us to honor the contracts.
Python began as a C-like scripting language. Its class mechanism and dependency injection are not widely adopted. However, good design pattern concepts should apply on large scale projects. The demo above has given a comparison of how to program to interfaces in two different ways. Since the class demo does not use any dependency injection, it has fewer lines of code as in swift
class module, but more lines on class initialization.
On the other hand, the python module way has encapsulated all dependencies in swift
module, all methods become immediately accessible after the import
. This is not the best example to prove using “traditional” python module is better. Because, if there are more dependencies coming from different domains or tiers, it would be difficult to achieve the same result (without writing a lot of wrappers or decorators).
If any module method has simple interface and dependencies, python module is just working fine (plus a little complication on config
in this demo); otherwise, if it ends up requiring many parameters (or dependencies), python class should be in a design consideration – after all, there must be a bigger reason of introducing class
into python than just pleasing OO developers.
Next topic (“Python Class in Design“) of this “Let Code Speak” series will use a more complicate task to demonstrate how python classes are used in OOP design. All source code in this demo are downloadable at here.
PicasaWeb full-size download
First of all, since Google+ was introduced, PicasaWeb has been automatically redirecting to plus.google.com
, which makes the URL longer and harder to remember, as well as missing some traditional PicasaWeb features/links (for example, “Linked Galleries”, “Sort”, Album Locations, and etc.). Note the transform from PicasaWeb to Google+ albums is from
- https://picasaweb.google.com/jason.zhuyx (where the last part is the email alias)
to
- https://plus.google.com/photos/103192840022832307624/albums (where the digits are Google+ id)
But other Google+ links (e.g. about/profile, posts) are actually starting after the id:
😦
Recently, Google adds a “Click here to go back to Picasa Web Albums” tab (in a light yellow background, for about a minute) over the top of Google+ albums. If the tab disappears, press F5
to refresh the page and get it back. The URL is using a “?noredirect=1
” query string to retain the Picasa Web.
Once in an album, each photo opened on PicasaWeb has an image URL (when you use “Save As”, or “Copy Image URL” link from context menu), like this:
However, the above link is only the image rendered in a browser. Depending on the system (e.g. PC, tablet, or phone device) and browser window, the size could be “s512”, “s640”, or “s800” (as the width of the image, or something like “w640-h480-no”, as dynamic rendering size, on Google+ page). For now the width can go up to 2560 (“s2560”) but never be the full size of the original uploaded picture.
In order to download the original photo, just simply change this “size path” to “d” or “s0-d“.
Caution: The changed link will pop up Download dialog instead of rendering the image. DO NOT use the URL in HTML <img src="" />
. And hope this protocol stays …
English Poetry
♥ By Robert Herrick (1591–1674)
Gather ye rosebuds while ye may,
Old Time is still a-flying:
And this same flower that smiles to-day
To-morrow will be dying.
The glorious lamp of heaven, the sun,
The higher he’s a-getting,
The sooner will his race be run,
And nearer he’s to setting.
♥ By Samuel Taylor Coleridge
The fair breeze blew, the white foam flew,
The furrow followed free;
We were the first that ever burst
Into that silent sea.
♥ Poetry Meter
In poetry, a unit of stressed and unstressed syllables is called a foot. For example, look at this line from Shakespeare: “No longer mourn for me when I am dead.” The rhythm is, “bah-BAH bah-BAH bah-BAH bah-BAH bah-BAH. We read it like this: “no LON-ger MOURN for ME when I am DEAD.” The type of foot Shakespeare used here is called an iamb (抑扬格,短长格).
- An iamb or an iambic foot has the rhythm bah-BAH. An unstressed syllable, then a stressed one. The iamb is the most common kind of foot in English poetry.
- The trochee or trochaic foot is the opposite of an iamb — the rhythm is BAH-bah, like the words “apple,” and “father.”
- The anapest or anapestic foot sounds like bah-bah-BAH, like the words “underneath” and “seventeen.”
- The dactyl or dactylic foot the opposite of an anapest — the rhythm is BAH-bah-bah,” like the the words “elephant” and “stepmother.”
- If there are two feet per line, it’s called dimeter. Here’s a sentence in trochaic dimeter: “Eat your dinner.” BAH-bah (1) BAH-bah (2).
- Three feet per line = trimeter. Here’s a sentence in iambic trimeter: “I eat the bread and cheese.” Bah-BAH (1) bah-BAH (2) bah-BAH (3).
- Four feet per line = tetrameter. Here’s a sentence in trochaic tetrameter: “Father ordered extra pizza.” BAH-bah (1) BAh-bah (2) BAH-bah (3) BAh-bah (4).
- Five feet per line = pentameter. Here’s a sentence in iambic pentameter: “I’ll toast the bread and melt a piece of cheese.” Bah-BAH (1) bah-BAH (2) bah-BAH (3) bah-BAH (4) bah-BAH (5).
- Six feet per line = hexameter or Alexandrine. A sentence in iambic hexameter: “I’ll toast the bread and melt a piece of cheese, okay?” Bah-BAH (1) bah-BAH (2) bah-BAH (3) bah-BAH (4) bah-BAH (5) bah-BAH (6).
- Seven feet per line = heptameter. …
♥ Meter and Rhythm
- None of us talk like robots. We give certain words and sounds more emphasis than others in a sentence, depending on a number of factors including the meaning of the words and our own personal speaking style. So not all of the stressed syllables have the same amount of stress, etc.
- We pause at the ends of ideas or the ends of sentences, even if these occur partway through a poetic line. So this creates a rhythmically variation. When the sentence ends or has a natural pause in the middle of a line of poetry, that’s called a caesura.
- Poets vary meter or make exceptions in order to create desired rhythmic effects.
See How to Write Poetry.
StartupCX
To improve customer experience, act like a startup:
- Serve more than you sell;
- Remember customers, not transactions;
- Don’t manage products, manage relationships;
- Operate for the long run, not for today’s closing stock price;
- Do the right thing, even if no one is watching.
- Be loyal to every employee who gives his or her all, especially when the economy gets tough.
- Takers has done enough damage. Put givers in charge of customer experience.
See video: StartupCX – Andy A./PeoplePerHour
Photography Notes
网上资料尽矣,此摘录笔记耳。
- 光圈值越小,光圈越大;光圈越大,景深越小,曝光值/通光量越大(或曰设定曝光值下,速度越快);
- 快门值越大,速度越快;速度越快,景深不变,曝光值/通光量越小【注:快门值小於一秒的分数值】
- 通光量减半,相当于快门速度翻倍,或光圈值递增/光圈缩小,即曝光值减一;
- 通光量翻倍,相当于快门速度减半,或光圈值递减/光圈变大,即曝光值加一;
- 感光度越大,相当于快门速度越慢,或曰在设定曝光值下,速度越快;
- 焦距值越大,景深越浅;物距越近,景深越浅。(决定景深的三要素:光圈,焦距,物距)
- 焦距值范围:10~14mm(鱼眼),14~35mm(广角),85mm(标准),120mm以上(长焦);注:用于非全幅数字相机时,焦距值要乘以系数(佳能1.6/尼康1.5)。
- 光圈值范围:全开(1),1.4,2,2.8,4,5.6,8,11,16,22,32(通光量依次倍减)
- 光圈优先表:设定感光度100,光圈f8时的快门=1/1000(阳光下白亮物体),1/500(全晴),1/250(薄雾),1/125(多云),1/60(阴天);
- 月夜曝光值:设定感光度100,光圈f5.6时的快门=1/60(满月);1秒(月牙);60分钟(满月月光,景中无月)
- 星轨曝光值:设定感光度1600,光圈f2.8时的快门=30秒以上(无月、野外;星星以北极星为轴心,每小时15度角旋转)
- 烟花曝光值:设定感光度100,光圈f8时的快门=1秒以上(使用快门线)
- HDR预置:下载 Photomatix Pro Presets
相机和镜头的购买指南
- Online tools: ShutterCount
- Review and comparision: Snapshot, Len’s Hero, DxoMark, kenrockwell.com, digitalcamerareview.com, dcresource.com, dpreview.com
- Shopping: adaroma, B&H Photo, ReallyRightStuff, 2filter
See: CSS Pattern Generators (as used for bullet image in this post)
Study Notes: Higher Education in Transition
Higher Education in Transition (4th Ed) - Beginnings John S. Brubacher & Willis Rudy Higher education in the United States has been molded and influenced by a variety of historical forces. On one hand, there are the patterns and traditions of higher learning which have been brought over from Western Europe. On the other, we find the native American conditions which have affected and modified the development of these transplanted institutions. Out of the interaction of these two essential elements and, most important, out of the growth of democracy in every area of American life, has developed a truly unique system of higher education. -------------------- English Influences --------------------