Javascript: Sync JSON Data

/**
 * Synchronize new data to original object and keep reference
 * Check object data type and traverse object tree to delete old key
 * as well as replace with new data:
 *
 * - if keyed data is directory (hash map) 'object', sync recursively
 * - else (including number, string, and array, etc), replace with new data
 *
 * @returns void.
 */
function syncObjectData(newData, orgData) {
  if (newData != null && orgData !== newData) {
    var sourceType = orgData.toType();
    var targetType = newData.toType();

    if (sourceType != targetType) {
      // the top level type must match

      var message = "Cannot update '" + sourceType + "' with different type " + targetType;
      throw new Error(message);
    } else if (sourceType != 'object') {
      // can only sync object type

      throw new Error("Does not support updating type of " + orgType);
    } else {
      for (var key in newData) {
        if (newData.hasOwnProperty(key)) {
          if (orgData.hasOwnProperty(key) && newData[key] != null) {
            var orgType = toType(orgData[key]);
            var newType = toType(newData[key]);

            if (newType == orgType && orgType == 'object' && orgData[key] != null) {
              syncObjectData(newData[key], orgData[key]);
            } else {
              orgData[key] = newData[key];
            }
          } else if (!orgData.hasOwnProperty(key)) {
            orgData[key] = newData[key];
          } else {
            delete orgData[key];
          }
        }
      }
      for (var key in orgData) {
        if (orgData.hasOwnProperty(key)) {
          if (!newData.hasOwnProperty(key) || newData[key] == null) {
            delete orgData[key];
          }
        }
      }
    }
  }
}

Here is the helper function to get name (string) of the exact type.

/**
 * Convert data type to string, e.g. 'undefined', 
 * or 'null', 'boolean', 'number', 'string', 'array', 'object', 'date', 'function', etc.
 * https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects
 */
function toType(variable) {
  return ({}).toString.call(variable).match(/\s([a-zA-Z]+)/)[1].toLowerCase();
}

Advertisements

Python class vs module

In object-oriented programming design, using class module is one solution to have dependencies injected and encapsulated in constructor (class initialization) with all public methods defining/implementing clear interfaces (contract). Although Python is not a typical OO language (nor a preference many developers using that way), it has class mechanism to accomplish such task. Let’s take a first look on python class.

1. Python class mechanism

The following demo is to write a swiftclient wrapper to access object store on OpenStack (cloud computing platform). Since swift client API methods requires certain dependencies (e.g. auth token, swift URL, and container name), it would be nice to wrap them in a class object instead of calling API methods with these in parameters. For example, ideal coding in application domain should look like this –

from keystone import get_auth_token
from swift import get_swift, Swift

# Use built-in swift_class module to get an instance of Swift
swift = get_swift()

# Otherwise, if dependencies (e.g. swift_url and container_name) in app domain
auth_token = get_auth_token()
swift = Swift(auth_token, swift_url, container_name)

# Once we have a Swift instance ...
afile = swift.get_file_contents('foo.txt')
files = swift.get_files_in_container()

Notice that if application settings need to be separated from swift module (most likely in a large-scale project), the class instantiation will be in application domain (where e.g. settings belong to); otherwise, the swift module could have a get_swift() function to construct a swift instance (like in this demo). The class module and unit tests are listed as below.

See common/swift.py (Swift class) –

# swift.py (Swift class)
import logging
import mimetypes
import sys

from swiftclient import client as swift_client
from swiftclient.exceptions import ClientException
from keystone import get_auth_token
from config import settings

logger = logging.getLogger(__name__)


# ================================================
# Swift instance initialization
# ================================================
def get_swift():
    """
    Get a Swift instance per application settings
    """
    auth_token = get_auth_token()
    container = settings('swift_container')
    swift_url = settings('swift_url')
    swift = Swift(auth_token, swift_url, container)
    return swift


# ================================================
# Swift class
# ================================================
class Swift(object):
    def __init__(self, auth_token, swift_url, container_name):
        """
        Initialize a Swift instance
        """
        self.auth_token = auth_token
        self.swift_url = swift_url
        self.container = container_name
        self.connection = self._get_connection()

    def _get_connection(self):
        try:
            return swift_client.Connection(
                preauthurl=self.swift_url,
                preauthtoken=self.auth_token,
                retries=5,
                auth_version='1',
                insecure=True)
        except Exception as e:
            err_message = "Exception raised initiating a swift connection."
            logger.exception(err_message)
            raise

    def check_container(self):
        """
        Determine if default container missing in Swift
        """
        try:
            headers, container_list = self.connection.get_account()
            for container in container_list:
                if container['name'] == self.container:
                    return True
            return False
        except Exception as e:
            err_message = "Exception raised on checking container exists."
            logger.exception(err_message)
            raise

    def ensure_container_exists(self):
        """
        Ensure default container exists in Swift.
        """
        # Determine if necessary container missing; if so, create it
        container_exists = self.check_container()
        if (not container_exists):
            try:
                response = {}
                self.connection.put_container(
                    self.container, response_dict=response)
            except Exception as e:
                err = "Exception on creating container {0}.".format(
                    self.container)
                logger.exception(err)
                raise

    def get_file_contents(self, file_name):
        """
        Function wrapper to perform 'get_object' call on Swift
        """
        try:
            response_dict = {}
            # Response from Swift:
            #     a tuple of (response headers, the object contents)
            #     The response headers will be a dict and all header names
            #     will be lowercase.
            response = self.connection.get_object(
                self.container,
                file_name,
                response_dict=response_dict)
            file_contents = response[1]
            return file_contents
        except Exception as e:
            err = "Exception on getting {0} from Swift.".format(file_name)
            logger.exception(err)
            raise

    def get_files_in_container(self):
        result = self.connection.get_container(
            self.container, full_listing=True)
        return result[1]

    def save_file(self, file_name, file_contents):
        """
        Function wrapper to perform 'put_object' call Swift
        """
        try:
            self.ensure_container_exists()
            response = {}
            # Example of response from put_object call -
            # {
            #     'status': 201,
            #     'headers': {
            #         'content-length': '0',
            #         'last-modified': 'Fri, 17 Jul 2015 04:43:56 GMT',
            #         'connection': 'keep-alive',
            #         'etag': 'd41d8cd98f00b204e9800998ecf8427e',
            #         'x-trans-id': 'txeddbca07d8e744deae343-0055a8880c',
            #         'date': 'Fri, 17 Jul 2015 04:43:57 GMT',
            #         'content-type': 'text/html; charset=UTF-8'},
            #     'reason': 'Created',
            #     'response_dicts': [{
            #         'status': 201,
            #         'headers': {
            #             'content-length': '0',
            #             'last-modified':
            #             'Fri, 17 Jul 2015 04:43:56 GMT',
            #             'connection': 'keep-alive',
            #             'etag': 'd41d8cd98f00b204e9800998ecf8427e',
            #             'x-trans-id': 'txeddbca07d8e744deae343-0055a8880c',
            #             'date': 'Fri, 17 Jul 2015 04:43:57 GMT',
            #             'content-type': 'text/html; charset=UTF-8'},
            #             'reason': 'Created'}]}
            self.connection.put_object(
                self.container,
                file_name,
                file_contents,
                content_length=sys.getsizeof(file_contents),
                content_type=mimetypes.guess_type(file_name, strict=True)[0],
                response_dict=response)
            return response
        except Exception as e:
            err_message = "Exception on saving file contents to Swift.\n"
            logger.exception(err_message)
            raise

See common/tests/swift_tests.py (Unit tests for Swift class) –

# swift_tests.py
import logging
import mock
import os
import StringIO
import sys
import tarfile
import unittest

from pyramid import testing
from mock import Mock, MagicMock, patch, mock_open
from swiftclient.exceptions import ClientException

from swift import Swift
logger = logging.getLogger(__name__)


class SwiftClassTests(unittest.TestCase):
    @patch('swiftclient.client.Connection')
    def setUp(self, mock_connection):
        self.config = testing.setUp()
        self.auth_token = Mock()
        self.account_data = ([{}], [{'name': 'container1'}])
        self.container = 'container1'
        self.container_data = ([{}], [{'name': 'file1'}, {'name': 'file2'}])
        self.swift_url = 'http://0.0.0.0'
        self.connection = Mock()
        # patch('common.swift.Swift.get_connection',
        #       return_value=self.connection
        mock_connection.return_value = self.connection
        self.swift = Swift(
            self.auth_token, self.swift_url, self.container)

    def tearDown(self):
        testing.tearDown()

    @patch('swiftclient.client.Connection')
    def test_constructor(self, mock_connection):
        self.assertEqual(self.swift.auth_token, self.auth_token)
        self.assertEqual(self.swift.swift_url, self.swift_url)
        self.assertEqual(self.swift.connection, self.connection)
        self.assertEqual(self.swift.container, self.container)

    @patch('swiftclient.client.Connection')
    def test_connection_exception(self, mock_connection):
        error_message = 'CONNECTION ERROR'
        mock_connection.side_effect = Exception(error_message)
        with self.assertRaises(Exception) as cm:
            result = Swift(self.auth_token, self.swift_url, self.container)
            self.assertEqual(str(cm.exception), error_message)

    def test_check_container_exception(self):
        name = 'foo'
        error_message = 'PUT CONTAINER ERROR'
        self.swift.connection.get_account.side_effect \
            = Exception(error_message)
        with self.assertRaises(Exception) as cm:
            result = self.swift.check_container()
            self.assertEqual(str(cm.exception), error_message)
            self.assertFalse(result)

    def test_check_container_when_false(self):
        self.swift.connection.get_account.return_value = self.account_data
        self.swift.container = 'dummy'
        result = self.swift.check_container()
        self.assertFalse(result)

    def test_check_container_when_true(self):
        self.swift.connection.get_account.return_value = self.account_data
        result = self.swift.check_container()
        self.assertTrue(result)

    def test_check_file_exists(self):
        self.swift.check_container = MagicMock()
        self.swift.check_container.return_value = True
        self.swift.connection.get_container = MagicMock()
        self.swift.connection.get_container.return_value = self.container_data
        result = self.swift.check_file_exists('fileX')
        self.assertFalse(result)
        result = self.swift.check_file_exists('file1')
        self.assertTrue(result)

    def test_check_file_exists_no_container(self):
        self.swift.check_container = MagicMock()
        self.swift.check_container.return_value = False
        result = self.swift.check_file_exists('filename')
        self.assertFalse(result)

    def test_ensure_container_exists(self):
        success = {'status': 200}

        def mock_put_success(container_name, response_dict):
            logger.debug(
                'Called with {0} {1}'.format(container_name, response_dict))
            response_dict['status'] = success['status']

        with patch.object(
                self.swift.connection, 'get_account',
                return_value=self.account_data):
            with patch.object(
                    self.swift.connection, 'put_container',
                    side_effect=mock_put_success) as mocked_put:
                self.swift.ensure_container_exists()
                mocked_put.assert_called()

    def test_ensure_container_exists_exception(self):
        error_message = 'PUT CONTAINER ERROR'

        with patch.object(
                self.swift.connection, 'get_account',
                return_value=self.account_data):
            with patch.object(
                    self.swift.connection, 'put_container',
                    side_effect=Exception(error_message)) as mocked_put:
                self.swift.check_container = MagicMock()
                self.swift.check_container.return_value = False
                with self.assertRaises(Exception) as cm:
                    self.swift.ensure_container_exists()
                    self.assertEqual(str(cm.exception), error_message)

    def test_get_file_contents(self):
        response = ([{}], '_filecontents')
        self.swift.connection.get_object = MagicMock()
        self.swift.connection.get_object.return_value = response
        result = self.swift.get_file_contents('file_name')
        self.assertEqual(result, response[1])

    def test_get_file_contents_exeption(self):
        error_message = 'Exception on get object'
        self.swift.connection.get_object = MagicMock()
        self.swift.connection.get_object.side_effect = Exception(error_message)
        with self.assertRaises(Exception) as cm:
            result = self.swift.get_file_contents('file_name')
            self.assertEqual(str(cm.exception), error_message)

    def test_get_files_in_container(self):
        self.swift.connection.get_container = MagicMock()
        self.swift.connection.get_container.return_value = self.container_data
        result = self.swift.get_files_in_container()
        self.assertEqual(result, self.container_data[1])

    @patch('mimetypes.guess_type')
    @patch('sys.getsizeof')
    def test_save_file_contents(self, mock_getsizeof, mock_guess_type):
        success = {'status': 200}

        def mock_put_success(
                container_name, file_name, contents,
                content_length, content_type, response_dict):
            response_dict['status'] = success['status']

        file_name = 'filename'
        contents = MagicMock()
        mock_getsizeof.return_value = 999
        mock_guess_type.return_value = ['filetype']
        with mock.patch.object(
                self.swift.connection, 'put_object',
                side_effect=mock_put_success) as mocked_put:
            self.swift.check_container = MagicMock()
            self.swift.check_container.return_value = True
            self.swift.save_file_contents(file_name, contents)
            mocked_put.assert_called_with(
                self.container,
                file_name, contents,
                content_length=999, content_type='filetype',
                response_dict=success)

    @patch('mimetypes.guess_type')
    @patch('sys.getsizeof')
    def test_save_file_contents_Exception(
            self, mock_getsizeof, mock_guess_type):
        file_name = 'filename'
        contents = MagicMock()
        mock_getsizeof.return_value = 999
        mock_guess_type.return_value = ['filetype']
        error_message = "SWIFT PUT OBJECT ERROR"
        self.swift.connection.put_object.side_effect = Exception(error_message)
        self.swift.check_container = MagicMock()
        self.swift.check_container.return_value = False
        with self.assertRaises(Exception) as cm:
            self.swift.save_file_contents(file_name, contents)
            self.assertEqual(str(cm.exception), error_message)

2. Python module

In use of Python class mechanism (as in above example), it takes a couple steps (of initialization, if no dependency injection used) before all class methods are accessible. Next, let’s see a comparison on how to make swift functions (more in a traditional python style) available right after import the module, in order to have fewer lines (under certain conditions, see discussion on next) of code as demonstrated below. The full source code (with config, keystone modules, and unit tests) are included at the end.

Beware of how @checks_config decorator is used, in swift_module.py, to guarantee properly instantiating a SwiftConfig if any swift method is called without one. One benefit using optional config=None parameter is to have some flexibility that a config can be either specified by the caller or created by swift module automatically. This makes swift module methods have same signatures (without config), comparing to class mechanism. But disadvantage (of such hiding) also appears that each auto-creation of a config by the decorator will have a different instance.

import swift_module as swift

# optionally to get a swift config first (see SwiftConfig in swift_module.py)
config = swift.get_swift_config()
# swift module functions are available after the import
a_file = swift.get_file_contents('foo.txt', config=config)
allist = swift.get_files_in_container(config=config)

See common/config.py

import logging
import pyramid

logger = logging.getLogger(__name__)


def checks_config(config_func):
    """
    Get decorator to use config_func as initiator for 'config' arg

    Keyword arguments:
    config_func -- a function to get proper configuration
    """
    def checks_config_decorator(original_func):
        """
        Call decorated original_func with checking its 'config' arg

        Keyword arguments:
        func -- original function to be decorated
        """
        def _arg_index_of(func, name):
            """
            Get the index of a named arg on a func call
            """
            import inspect
            argspec = inspect.getargspec(func)
            for i in range(len(argspec[0])):
                if (argspec[0][i] == name):
                    logger.debug("argspec[0][{0}]=={1}".format(i, name))
                    return i
            return -1

        def _checks_config_wrapper(*args, **kwargs):
            """
            Check 'config' arg before calling original_func
            Call specified config_func if 'config' arg value is None.
            """
            arg_idx = _arg_index_of(original_func, 'config')
            has_arg = 0 <= arg_idx and arg_idx < len(args)
            arg_cfg = args[arg_idx] if (has_arg) else None
            kwa_cfg = kwargs.get('config')
            if (kwa_cfg is None and arg_cfg is None):
                # logger.debug('Getting config by {0}'.format(config_func))
                kwargs['config'] = config_func()
            return original_func(*args, **kwargs)

        # calls the original function with checking proper configuration
        return _checks_config_wrapper
    # returns a decorated function
    return checks_config_decorator


def settings(item):
    """
    Get a reference to the settings in the .ini file
    """
    registry = pyramid.threadlocal.get_current_registry()
    return registry.settings.get(item, None)

See common/keystone.py

from config import settings
from keystoneclient.v2_0 import client as keystone_client
from logging import getLogger
logger = getLogger(__name__)


def get_auth_token():
    """
    Get an auth token from Keystone.
    """
    try:
        keystone = keystone_client.Client(
            username=settings('cloud_username'),
            password=settings('cloud_password'),
            tenant_name=settings('cloud_project_name'),
            auth_url=settings('cloud_auth_url'),
            insecure=True)
        return keystone.auth_ref['token']['id']
    except Exception as e:
        logger.error(
            "Exception authenticating against Keystone")
        logger.exception("Details: {0}".format(e))
        raise

See common/swift_module.py

# swift_module.py
import logging
import mimetypes
import sys

from config import checks_config, settings
from keystone import get_auth_token
from swiftclient import client as swift_client
from swiftclient.exceptions import ClientException

logger = logging.getLogger(__name__)


# ================================================
# Swift configuration class
# ================================================
class SwiftConfig(object):
    def __init__(self, auth_token, swift_url, container_name):
        """
        Initialize a Swift configuration instance
        """
        self.auth_token = auth_token
        self.swift_url = swift_url
        self.container = container_name
        self.connection = self._get_connection()

    def _get_connection(self):
        """
        Get a connection to Swift object store
        """
        try:
            return swift_client.Connection(
                preauthurl=self.swift_url,
                preauthtoken=self.auth_token,
                retries=5,
                auth_version='1',
                insecure=True)
        except Exception as e:
            err_message = "Exception raised initiating a swift connection."
            logger.exception(err_message)
            raise

# ToDo [zhuyux]: considering singleton for SwiftConfig instance
_swift_config_singleton = None

# ================================================
# Swift configuration initialization
# ================================================
def get_swift_config():
    """
    Get a SwiftConfig instance per application settings
    """
    auth_token = get_auth_token()
    container = settings('swift_container')
    swift_url = settings('swift_url')
    swift_cfg = SwiftConfig(auth_token, swift_url, container)
    return swift_cfg


def _get_config():
    """
    This is a fixed/non-mockable func pointer for @checks_config decorator
    """
    # logger.debug('get_swift_config={0}'.format(get_swift_config))
    return get_swift_config()

# ================================================
# Swift module interfaces
# ================================================

@checks_config(config_func=_get_config)
def check_container(config=None):
    """
    Check if default container exists in Swift

    Keyword arguments:
    config -- an instance of SwiftConfig (optional, default None)
    """
    try:
        logger.debug('Checking container {0}'.format(config.container))
        headers, container_list = config.connection.get_account()
        for container in container_list:
            logger.debug("--- container: {0}".format(container['name']))
            if (container['name'] == config.container):
                logger.debug('--- found {0}'.format(config.container))
                return False
        logger.debug('--- missing container {0}'.format(config.container))
        return True
    except Exception as e:
        err_message = "Exception raised on checking container exists."
        logger.exception(err_message)
        raise


@checks_config(config_func=_get_config)
def check_file_exists(file_name, config=None):
    """
    Check if specified file exists in Swift store

    Keyword arguments:
    file_name -- the name of the file to be checked in Swift store
    config -- an instance of SwiftConfig (optional, default None)
    """
    if (check_container(config=config)):
        files = get_files_in_container(config=config)
        for file in files:
            if (file['name'] == file_name):
                return True
    return False


@checks_config(config_func=_get_config)
def ensure_container_exists(config=None):
    """
    Ensure default container exists in Swift.

    Keyword arguments:
    config -- an instance of SwiftConfig (optional, default None)
    """
    container_exists = check_container(config=config)
    if (not container_exists):
        try:
            response = {}
            config.connection.put_container(
                config.container, response_dict=response)
            logger.debug(
                "--- Container {0} created".format(config.container))
            logger.debug("--- Response {0}".format(response))
        except Exception as e:
            err = "Exception on creating container {0}.".format(
                config.container)
            logger.exception(err)
            raise


@checks_config(config_func=_get_config)
def get_file_contents(file_name, config=None):
    """
    Function wrapper to perform 'get_object' call on Swift

    Keyword arguments:
    file_name -- the name of the file in Swift store
    config -- an instance of SwiftConfig (optional, default None)
    """
    try:
        response_dict = {}
        # Response from Swift:
        #   a tuple of (response headers, the object contents)
        #   The response headers will be a dict and all header names
        #   will be in lower case.
        response = config.connection.get_object(
            config.container,
            file_name,
            response_dict=response_dict)
        file_contents = response[1]
        return file_contents
    except Exception as e:
        err = "Exception on getting {0} from Swift.".format(file_name)
        logger.exception(err)
        raise


@checks_config(config_func=_get_config)
def get_files_in_container(config=None):
    """
    Get info of all files in default Swift container

    Keyword arguments:
    config -- an instance of SwiftConfig (optional, default None)
    """
    result = config.connection.get_container(
        config.container, full_listing=True)
    return result[1]


@checks_config(config_func=_get_config)
def save_file(file_name, file_contents, config=None):
    """
    Function wrapper to perform 'put_object' call Swift

    Keyword arguments:
    file_name -- the name of the file to be saved
    file_contents -- the contents of the file to be saved in Swift store
    config -- an instance of SwiftConfig (optional, default None)
    """
    try:
        # Ensure the default container exists
        ensure_container_exists(config=config)
        # Push the file contents to Swift
        response = {}
        config.connection.put_object(
            config.container,
            file_name,
            file_contents,
            content_length=sys.getsizeof(file_contents),
            content_type=mimetypes.guess_type(file_name, strict=True)[0],
            response_dict=response)
        return response
    except Exception as e:
        err = "Exception on saving file contents to Swift.\n"
        logger.exception(err)
        raise

See common/tests/swift_module_tests.py

import logging
import mock
import os
import StringIO
import sys
import tarfile
import unittest

import common.swift_module as swift

from pyramid import testing
from mock import Mock, MagicMock, patch, mock_open
from swiftclient.exceptions import ClientException

logger = logging.getLogger(__name__)


class SwiftTests(unittest.TestCase):
    @patch('common.swift_module.get_auth_token')
    @patch('swiftclient.client.Connection')
    def setUp(self, mock_swift_connection, mock_get_auth_token):
        self.config = testing.setUp()
        self.auth_token = MagicMock()
        self.account_data = ([{}], [{'name': 'container1'}])
        self.container = 'container1'
        self.container_data = ([{}], [{'name': 'file1'}, {'name': 'file2'}])
        self.swift_url = 'http://0.0.0.0'
        self.setting = 'dummy setting'
        self.connection = Mock()
        mock_swift_connection.return_value = self.connection
        mock_get_auth_token.return_value = self.auth_token

        self.swift_cfg = swift.SwiftConfig(
            self.auth_token, self.swift_url, self.container)
        self.swift_cfg.connection.get_account.return_value = self.account_data

    def tearDown(self):
        testing.tearDown()

    @patch('swiftclient.client.Connection')
    def test_swift_config(self, mock_connection):
        self.assertEqual(self.swift_cfg.auth_token, self.auth_token)
        self.assertEqual(self.swift_cfg.swift_url, self.swift_url)
        self.assertEqual(self.swift_cfg.connection, self.connection)
        self.assertEqual(self.swift_cfg.container, self.container)

    @patch('swiftclient.client.Connection')
    def test_swift_config_exception(self, mock_connection):
        error_message = 'CONNECTION ERROR'
        mock_connection.side_effect = Exception(error_message)
        with self.assertRaises(Exception) as cm:
            result = swift.SwiftConfig(
                self.auth_token, self.swift_url, self.container)
            self.assertEqual(str(cm.exception), error_message)

    def test_check_container_exception(self):
        error_message = 'GET ACCOUNT ERROR'
        self.swift_cfg.connection.get_account.side_effect \
            = Exception(error_message)
        with self.assertRaises(Exception) as cm:
            result = swift.check_container(config=self.swift_cfg)
            self.assertEqual(str(cm.exception), error_message)
            self.assertFalse(result)

    @patch('common.swift_module.get_swift_config')
    def test_check_container_when_false(self, mock_get_config):
        mock_get_config.return_value = self.swift_cfg
        self.swift_cfg.container = '-=#=-dummy-=#=-'
        result = swift.check_container()
        self.assertFalse(result)

    @patch('common.swift_module.get_auth_token')
    @patch('swiftclient.client.Connection')
    def test_check_container_when_true(
            self, mock_swift_connection, mock_get_auth_token):
        mock_swift_connection.return_value = self.connection
        mock_get_auth_token.return_value = self.auth_token
        result = swift.check_container(config=self.swift_cfg)
        self.assertTrue(result)
        self.swift_cfg.container = '-=#=-dummy-=#=-'
        result = swift.check_container(config=self.swift_cfg)
        self.assertFalse(result)

    @patch('common.swift_module.get_swift_config')
    @patch('common.swift_module.check_container')
    def test_check_file_exists(self, mock_check_container, mock_get_config):
        mock_check_container.return_value = True
        mock_get_config.return_value = self.swift_cfg
        self.swift_cfg.connection.get_container = MagicMock()
        self.swift_cfg.connection.get_container.return_value = \
            self.container_data
        result = swift.check_file_exists('fileX')
        self.assertFalse(result)
        result = swift.check_file_exists('file1')
        self.assertTrue(result)

    @patch('common.swift_module.check_container')
    def test_check_file_exists_no_container(self, mock_check_container):
        mock_check_container.return_value = False
        result = swift.check_file_exists('filename', config=self.swift_cfg)
        self.assertFalse(result)

    def test_ensure_container_exists(self):
        success = {'status': 200}

        def mock_put_success(container_name, response_dict):
            logger.debug(
                'Called with {0} {1}'.format(container_name, response_dict))
            response_dict['status'] = success['status']

        with patch.object(
                self.swift_cfg.connection, 'get_account',
                return_value=self.account_data):
            with patch.object(
                    self.swift_cfg.connection, 'put_container',
                    side_effect=mock_put_success) as mocked_put:
                swift.ensure_container_exists(config=self.swift_cfg)
                mocked_put.assert_called()

    def test_ensure_container_exists_exception(self):
        error_message = 'PUT CONTAINER ERROR'

        with patch.object(
                self.swift_cfg.connection, 'get_account',
                return_value=self.account_data):
            with patch.object(
                    self.swift_cfg.connection, 'put_container',
                    side_effect=Exception(error_message)) as mocked_put:
                with self.assertRaises(Exception) as cm:
                    import common.swift
                    swift.ensure_container_exists(config=self.swift_cfg)
                    self.assertEqual(str(cm.exception), error_message)

    def test_get_file_contents(self):
        import common.swift
        response = ([{}], '_filecontents')
        self.swift_cfg.connection.get_object = MagicMock()
        self.swift_cfg.connection.get_object.return_value = response
        result = swift.get_file_contents('file_name', config=self.swift_cfg)
        self.assertEqual(result, response[1])

    def test_get_file_contents_exeption(self):
        error_message = 'Exception on get object'
        self.swift_cfg.connection.get_object = MagicMock()
        self.swift_cfg.connection.get_object.side_effect = Exception(error_message)
        with self.assertRaises(Exception) as cm:
            swift.get_file_contents('file_name', config=self.swift_cfg)
            self.assertEqual(str(cm.exception), error_message)

    def test_get_files_in_container(self):
        self.swift_cfg.connection.get_container = MagicMock()
        self.swift_cfg.connection.get_container.return_value = \
            self.container_data
        result = swift.get_files_in_container(config=self.swift_cfg)
        self.assertEqual(result, self.container_data[1])

    @patch('common.config.settings')
    @patch('common.swift_module.get_auth_token')
    @patch('swiftclient.client.Connection')
    def test_get_swift_config(
            self, mock_connection, mock_get_auth_token, mock_settings):
        mock_connection.return_value = self.connection
        mock_get_auth_token.return_value = self.auth_token
        mock_settings.return_value = self.setting
        result = swift.get_swift_config()
        self.assertEqual(result.auth_token, self.auth_token)
        self.assertEqual(result.connection, self.connection)
        self.assertEqual(result.container, self.setting)
        self.assertEqual(result.swift_url, self.setting)

    @patch('mimetypes.guess_type')
    @patch('sys.getsizeof')
    def test_save_file(self, mock_getsizeof, mock_guess_type):
        success = {'status': 200}

        def mock_put_success(
                container_name, file_name, contents,
                content_length, content_type, response_dict):
            response_dict['status'] = success['status']

        mock_getsizeof.return_value = 999
        mock_guess_type.return_value = ['filetype']
        with mock.patch.object(
                self.swift_cfg.connection, 'put_object',
                side_effect=mock_put_success) as mocked_put:
            import common.swift
            swift.check_container = MagicMock()
            swift.check_container.return_value = True
            filename = 'filename'
            contents = MagicMock()
            swift.save_file(filename, contents, config=self.swift_cfg)
            mocked_put.assert_called_with(
                self.container,
                filename, contents,
                content_length=999, content_type='filetype',
                response_dict=success)

    @patch('mimetypes.guess_type')
    @patch('sys.getsizeof')
    def test_save_file_Exception(
            self, mock_getsizeof, mock_guess_type):
        import common.swift
        mock_getsizeof.return_value = 999
        mock_guess_type.return_value = ['filetype']
        error_message = "SWIFT PUT OBJECT ERROR"
        self.swift_cfg.connection.put_object.side_effect = \
            Exception(error_message)
        swift.check_container = MagicMock()
        swift.check_container.return_value = False
        with self.assertRaises(Exception) as cm:
            filename = 'filename'
            contents = MagicMock()
            swift.save_file(filename, contents, config=self.swift_cfg)
            self.assertEqual(str(cm.exception), error_message)

Summary

Programmers like to write simple code (to make it clear and easy to understand, thus easier to share, test, and maintain). Sure there is always a cost of effort. In OO design (either with C++, Java, or C#), dependency injection pattern has been applied, not only to write less and cleaner code, but also to support the concept of “programming to interfaces, not implementations” – forcing us to honor the contracts.

Python began as a C-like scripting language. Its class mechanism and dependency injection are not widely adopted. However, good design pattern concepts should apply on large scale projects. The demo above has given a comparison of how to program to interfaces in two different ways. Since the class demo does not use any dependency injection, it has fewer lines of code as in swift class module, but more lines on class initialization.

On the other hand, the python module way has encapsulated all dependencies in swift module, all methods become immediately accessible after the import. This is not the best example to prove using “traditional” python module is better. Because, if there are more dependencies coming from different domains or tiers, it would be difficult to achieve the same result (without writing a lot of wrappers or decorators).

If any module method has simple interface and dependencies, python module is just working fine (plus a little complication on config in this demo); otherwise, if it ends up requiring many parameters (or dependencies), python class should be in a design consideration – after all, there must be a bigger reason of introducing class into python than just pleasing OO developers.

Next topic (“Python Class in Design“) of this “Let Code Speak” series will use a more complicate task to demonstrate how python classes are used in OOP design. All source code in this demo are downloadable at here.

PicasaWeb full-size download

First of all, since Google+ was introduced, PicasaWeb has been automatically redirecting to plus.google.com, which makes the URL longer and harder to remember, as well as missing some traditional PicasaWeb features/links (for example, “Linked Galleries”, “Sort”, Album Locations, and etc.). Note the transform from PicasaWeb to Google+ albums is from

• https://picasaweb.google.com/jason.zhuyx (where the last part is the email alias)

to

• https://plus.google.com/photos/103192840022832307624/albums (where the digits are Google+ id)

But other Google+ links (e.g. about/profile, posts) are actually starting after the id:

• https://plus.google.com/103192840022832307624/

😦

Recently, Google adds a “Click here to go back to Picasa Web Albums” tab (in a light yellow background, for about a minute) over the top of Google+ albums. If the tab disappears, press F5 to refresh the page and get it back. The URL is using a “?noredirect=1” query string to retain the Picasa Web.

• https://picasaweb.google.com/jason.zhuyx?noredirect=1

Once in an album, each photo opened on PicasaWeb has an image URL (when you use “Save As”, or “Copy Image URL” link from context menu), like this:

• https://lh5.googleusercontent.com/-…/…/s800/20140718_1911_004.jpg

However, the above link is only the image rendered in a browser. Depending on the system (e.g. PC, tablet, or phone device) and browser window, the size could be “s512”, “s640”, or “s800” (as the width of the image, or something like “w640-h480-no”, as dynamic rendering size, on Google+ page). For now the width can go up to 2560 (“s2560”) but never be the full size of the original uploaded picture.

In order to download the original photo, just simply change this “size path” to “d” or “s0-d“.

Caution: The changed link will pop up Download dialog instead of rendering the image. DO NOT use the URL in HTML <img src="" />. And hope this protocol stays …

English Poetry

This is from my study notes.

♥ By Robert Herrick (1591–1674)

Gather ye rosebuds while ye may,
Old Time is still a-flying:
And this same flower that smiles to-day

To-morrow will be dying.
The glorious lamp of heaven, the sun,
The higher he’s a-getting,
The sooner will his race be run,
And nearer he’s to setting.

♥ By Samuel Taylor Coleridge

The fair breeze blew, the white foam flew,
The furrow followed free;
We were the first that ever burst
Into that silent sea.

♥ Poetry Meter

In poetry, a unit of stressed and unstressed syllables is called a foot. For example, look at this line from Shakespeare: “No longer mourn for me when I am dead.” The rhythm is, “bah-BAH bah-BAH bah-BAH bah-BAH bah-BAH. We read it like this: “no LON-ger MOURN for ME when I am DEAD.” The type of foot Shakespeare used here is called an iamb （抑扬格，短长格）.

• An iamb or an iambic foot has the rhythm bah-BAH. An unstressed syllable, then a stressed one. The iamb is the most common kind of foot in English poetry.
• The trochee or trochaic foot is the opposite of an iamb — the rhythm is BAH-bah, like the words “apple,” and “father.”
• The anapest or anapestic foot sounds like bah-bah-BAH, like the words “underneath” and “seventeen.”
• The dactyl or dactylic foot the opposite of an anapest — the rhythm is BAH-bah-bah,” like the the words “elephant” and “stepmother.”
• If there are two feet per line, it’s called dimeter. Here’s a sentence in trochaic dimeter: “Eat your dinner.” BAH-bah (1) BAH-bah (2).
• Three feet per line = trimeter. Here’s a sentence in iambic trimeter: “I eat the bread and cheese.” Bah-BAH (1) bah-BAH (2) bah-BAH (3).
• Four feet per line = tetrameter. Here’s a sentence in trochaic tetrameter: “Father ordered extra pizza.” BAH-bah (1) BAh-bah (2) BAH-bah (3) BAh-bah (4).
• Five feet per line = pentameter. Here’s a sentence in iambic pentameter: “I’ll toast the bread and melt a piece of cheese.” Bah-BAH (1) bah-BAH (2) bah-BAH (3) bah-BAH (4) bah-BAH (5).
• Six feet per line = hexameter or Alexandrine. A sentence in iambic hexameter: “I’ll toast the bread and melt a piece of cheese, okay?” Bah-BAH (1) bah-BAH (2) bah-BAH (3) bah-BAH (4) bah-BAH (5) bah-BAH (6).
• Seven feet per line = heptameter. …

♥ Meter and Rhythm

• None of us talk like robots. We give certain words and sounds more emphasis than others in a sentence, depending on a number of factors including the meaning of the words and our own personal speaking style. So not all of the stressed syllables have the same amount of stress, etc.
• We pause at the ends of ideas or the ends of sentences, even if these occur partway through a poetic line. So this creates a rhythmically variation. When the sentence ends or has a natural pause in the middle of a line of poetry, that’s called a caesura.
• Poets vary meter or make exceptions in order to create desired rhythmic effects.

See How to Write Poetry.

StartupCX

To improve customer experience, act like a startup:

• Serve more than you sell;
• Remember customers, not transactions;
• Don’t manage products, manage relationships;
• Operate for the long run, not for today’s closing stock price;
• Do the right thing, even if no one is watching.
• Be loyal to every employee who gives his or her all, especially when the economy gets tough.
• Takers has done enough damage. Put givers in charge of customer experience.

See video: StartupCX – Andy A./PeoplePerHour

Photography Notes

网上资料尽矣，此摘录笔记耳。

• 光圈值越小，光圈越大；光圈越大，景深越小，曝光值／通光量越大（或曰设定曝光值下，速度越快）；
• 快门值越大，速度越快；速度越快，景深不变，曝光值／通光量越小【注：快门值小於一秒的分数值】
• 通光量减半，相当于快门速度翻倍，或光圈值递增／光圈缩小，即曝光值减一；
• 通光量翻倍，相当于快门速度减半，或光圈值递减／光圈变大，即曝光值加一；
• 感光度越大，相当于快门速度越慢，或曰在设定曝光值下，速度越快；
• 焦距值越大，景深越浅；物距越近，景深越浅。（决定景深的三要素：光圈，焦距，物距）
• 焦距值范围：１０～１４ｍｍ（鱼眼），１４～３５ｍｍ（广角），８５ｍｍ（标准），１２０ｍｍ以上（长焦）；注：用于非全幅数字相机时，焦距值要乘以系数（佳能１.６／尼康１.５）。
• 光圈值范围：全开（１），１.４，２，２.８，４，５.６，８，１１，１６，２２，３２（通光量依次倍减）
• 光圈优先表：设定感光度１００，光圈ｆ８时的快门＝１／１０００（阳光下白亮物体），１／５００（全晴），１／２５０（薄雾），１／１２５（多云），１／６０（阴天）；
• 月夜曝光值：设定感光度１００，光圈ｆ５.６时的快门＝１／６０（满月）；１秒（月牙）；６０分钟（满月月光，景中无月）
• 星轨曝光值：设定感光度１６００，光圈ｆ２.８时的快门＝３０秒以上（无月、野外；星星以北极星为轴心，每小时１５度角旋转）
• 烟花曝光值：设定感光度１００，光圈ｆ８时的快门＝１秒以上（使用快门线）
• ＨＤＲ预置：下载 Photomatix Pro Presets

相机和镜头的购买指南

• Online tools: ShutterCount
• Review and comparision: Snapshot, Len’s Hero, DxoMark, kenrockwell.com, digitalcamerareview.com, dcresource.com, dpreview.com
• Shopping: adaroma, B&H Photo, ReallyRightStuff, 2filter

See: CSS Pattern Generators (as used for bullet image in this post)

• CodePen
• GeneratePattern.com
• BLog.IconFinder.com
• Patternify.com

Study Notes: Higher Education in Transition

Higher Education in Transition (4th Ed) - Beginnings
John S. Brubacher & Willis Rudy

　　Higher education in the United States has been molded and influenced by a variety of historical forces. On one hand, there are the patterns and traditions of higher learning which have been brought over from Western Europe. On the other, we find the native American conditions which have affected and modified the development of these transplanted institutions. Out of the interaction of these two essential elements and, most important, out of the growth of democracy in every area of American life, has developed a truly unique system of higher education.

--------------------
English Influences
--------------------