What is Unicode

To understand what Unicode is, first we need to understand basic encoding scheme called ASCII. Essentially ASSCII encodes every character in 7-bits, that means it can represent only 128 possible set of characters. Unicode is just a massive version of the ASCII table, which can represent 1,114,112 possible characters. In fact first 128 characters of Unicode table correspong precisely to the ASCII characters. That makes Unicode backward compatible to ASCII. For more detail description of Unicode follow this link.

But remember that Unicode is not an encoding scheme, it's just a collection of code points representing characters and symbols. Unicode has various encoding sheme like UTF-8 (most commonly used), UTF-16 and UTF-32 are used for representing Unicode characterss as binary data.

Every character in the Unicode table is mapped to something called unicode code point which is represented in hexadecimal (e.g. code point for ཀ is \u0f40). Here is the unicode code point in python.

'ཀ'.encode('unicode_escape')  # get unicode code point

b'\\u0f40'

int(0x0f40) # 0x denotes hexadecimal

3904

ord('ཀ') # check for integer value.

3904

Every language is given a specific unicode code point range. For eg:

0020-007f (32-127) for basic latin, similary to ASCII code point.
0f00-0fff (3840-4095) for Tibetan

int(0x0020), int(0x007f), int(0x0f00), int(0x0fff)

(32, 127, 3840, 4095)

UTF-8 - Great Hack of Unicode

We obviously know that for code point of a character larger than numeral 255 can't be represented with one byte (or 8-bits). It will need more than 1 bytes. So here comes UFT-16 which represent every character in two bytes and if code point is larger than that we need have UTF-32 wich represent every character in 4 bytes.

But there some big issues with UTF-16 and UTF-32:

Waste lots of memory, lots of bytes with all 0s.
Produces a 0x00 (i.e 00000000), which in many old computer interpreted as end of the string of characters.
Not compatible with ASCII.
Required BOM

Luckily UTF-8 solves all these problems with it's wonderful hack. The hack is how it can represent all other non ASCII characters.

UTF-8 Character Representation

UTF-8 was designed for backwrad compatibility with ASCII. So, the first 128 characters are ASCII characters with exact same order. Since unicode character code point order below the 255 requires extra bytes which makes UTF-8 a variable-length encoding scheme and it can encode code points in one to four bytes.

But the questions remain, how it represent unicode character which requires more than one byte while still avoiding eight consecutive 0s ? How it can specify the start of a characters ?. Well, let me explain it's unicode character representation along with an example.

Consider we are encoding first letter of Tibetan alphabet, ཀ:

The Unicode code point for ཀ is U+0F40
Hexadecimal 0F40 is binary 00001111 01000000
In UTF-8 encoding, first four bits tells how many bytes required to represent the given code point. According UTF-8 scheme table, ཀ will need three bytes. So, the those four bits will have three 1s (like 1110...).
Then four most significant bits of the code point are stored in the remaining low order four bits of the first byte (like 11100000).
All continuation bytes contains six bits from the code point because 10 is stored in the high order two bits to mark it as a continuation byte. It also avoids eight consecutive 0s. So, the remainingg bits of code point is placed like this 10111101 10000000.

So ཀ in UTF-8 binary is 11100000 10111101 10000000. For more about UTF-8 follow this link

Calulating Numeral value of a Unicode code point in python

'ཀ'.encode('utf-8')  # in hexadecimal

b'\xe0\xbd\x80'

" ".join(f"{i:08b}" for i in (0xe0, 0xbd, 0x80))  # convert hex to binary

'11100000 10111101 10000000'

As you can see here, we got the exact same binary value of ཀ as above.

Extract code point bits from UTF-8 bits:

Steps:

Seperate out code point bits, 11100000 10111101 10000000
We got the code point bits of ཀ, 00001111 01000000

convert extracted code point bits to int

int('0000111101000000', 2)

3904

Compare it with output of python's ord function

int('0000111101000000', 2) == ord('ཀ')

True

Fun facts

95% of web pages in the internet uses UTF-8 encoding at time of wiriting this blog.

About

This notebook is a demonstration of some of capabilities of fastpages with notebooks.

With fastpages you can save your jupyter notebooks into the _notebooks folder at the root of your repository, and they will be automatically be converted to Jekyll compliant blog posts!

Front Matter

The first cell in your Jupyter Notebook or markdown blog post contains front matter. Front matter is metadata that can turn on/off options in your Notebook. It is formatted like this:

# "My Title"
> "Awesome summary"

- toc:true- branch: master- badges: true- comments: true
- author: Hamel Husain & Jeremy Howard
- categories: [fastpages, jupyter]

Setting toc: true will automatically generate a table of contents
Setting badges: true will automatically include GitHub and Google Colab links to your notebook.
Setting comments: true will enable commenting on your blog post, powered by utterances.

The title and description need to be enclosed in double quotes only if they include special characters such as a colon. More details and options for front matter can be viewed on the front matter section of the README.

Markdown Shortcuts

A #hide comment at the top of any code cell will hide both the input and output of that cell in your blog post.

A #hide_input comment at the top of any code cell will only hide the input of that cell.

The comment #hide_input was used to hide the code that produced this.

put a #collapse-hide flag at the top of any cell if you want to hide that cell by default, but give the reader the option to show it:

#collapse-hide
import pandas as pd
import altair as alt

put a #collapse-show flag at the top of any cell if you want to show that cell by default, but give the reader the option to hide it:

#collapse-show
cars = 'https://vega.github.io/vega-datasets/data/cars.json'
movies = 'https://vega.github.io/vega-datasets/data/movies.json'
sp500 = 'https://vega.github.io/vega-datasets/data/sp500.csv'
stocks = 'https://vega.github.io/vega-datasets/data/stocks.csv'
flights = 'https://vega.github.io/vega-datasets/data/flights-5k.json'

Interactive Charts With Altair

Charts made with Altair remain interactive. Example charts taken from this repo, specifically this notebook.

Example 1: DropDown

# single-value selection over [Major_Genre, MPAA_Rating] pairs
# use specific hard-wired values as the initial selected values
selection = alt.selection_single(
    name='Select',
    fields=['Major_Genre', 'MPAA_Rating'],
    init={'Major_Genre': 'Drama', 'MPAA_Rating': 'R'},
    bind={'Major_Genre': alt.binding_select(options=genres), 'MPAA_Rating': alt.binding_radio(options=mpaa)}
)
  
# scatter plot, modify opacity based on selection
alt.Chart(movies).mark_circle().add_selection(
    selection
).encode(
    x='Rotten_Tomatoes_Rating:Q',
    y='IMDB_Rating:Q',
    tooltip='Title:N',
    opacity=alt.condition(selection, alt.value(0.75), alt.value(0.05))
)

Example 2: Tooltips

alt.Chart(movies).mark_circle().add_selection(
    alt.selection_interval(bind='scales', encodings=['x'])
).encode(
    x='Rotten_Tomatoes_Rating:Q',
    y=alt.Y('IMDB_Rating:Q', axis=alt.Axis(minExtent=30)), # use min extent to stabilize axis title placement
    tooltip=['Title:N', 'Release_Date:N', 'IMDB_Rating:Q', 'Rotten_Tomatoes_Rating:Q']
).properties(
    width=600,
    height=400
)

Example 3: More Tooltips

# select a point for which to provide details-on-demand
label = alt.selection_single(
    encodings=['x'], # limit selection to x-axis value
    on='mouseover',  # select on mouseover events
    nearest=True,    # select data point nearest the cursor
    empty='none'     # empty selection includes no data points
)

# define our base line chart of stock prices
base = alt.Chart().mark_line().encode(
    alt.X('date:T'),
    alt.Y('price:Q', scale=alt.Scale(type='log')),
    alt.Color('symbol:N')
)

alt.layer(
    base, # base line chart
    
    # add a rule mark to serve as a guide line
    alt.Chart().mark_rule(color='#aaa').encode(
        x='date:T'
    ).transform_filter(label),
    
    # add circle marks for selected time points, hide unselected points
    base.mark_circle().encode(
        opacity=alt.condition(label, alt.value(1), alt.value(0))
    ).add_selection(label),

    # add white stroked text to provide a legible background for labels
    base.mark_text(align='left', dx=5, dy=-5, stroke='white', strokeWidth=2).encode(
        text='price:Q'
    ).transform_filter(label),

    # add text labels for stock prices
    base.mark_text(align='left', dx=5, dy=-5).encode(
        text='price:Q'
    ).transform_filter(label),
    
    data=stocks
).properties(
    width=700,
    height=400
)

Data Tables

You can display tables per the usual way in your blog:

movies = 'https://vega.github.io/vega-datasets/data/movies.json'
df = pd.read_json(movies)
# display table with pandas
df[['Title', 'Worldwide_Gross', 
    'Production_Budget', 'Distributor', 'MPAA_Rating', 'IMDB_Rating', 'Rotten_Tomatoes_Rating']].head()

Images

Local Images

You can reference local images and they will be copied and rendered on your blog automatically. You can include these with the following markdown syntax:

![](my_icons/fastai_logo.png)

Remote Images

Remote images can be included with the following markdown syntax:

![](https://image.flaticon.com/icons/svg/36/36686.svg)

Animated Gifs

Animated Gifs work, too!

![](https://upload.wikimedia.org/wikipedia/commons/7/71/ChessPawnSpecialMoves.gif)

Captions

You can include captions with markdown images like this:

![](https://www.fast.ai/images/fastai_paper/show_batch.png "Credit: https://www.fast.ai/2020/02/13/fastai-A-Layered-API-for-Deep-Learning/")

Other Elements

Tweetcards

Typing > twitter: https://twitter.com/jakevdp/status/1204765621767901185?s=20 will render this:

Altair 4.0 is released! https://t.co/PCyrIOTcvv
Try it with:

pip install -U altair

The full list of changes is at https://t.co/roXmzcsT58 ...read on for some highlights. pic.twitter.com/vWJ0ZveKbZ
— Jake VanderPlas (@jakevdp) December 11, 2019

Youtube Videos

Typing > youtube: https://youtu.be/XfoYk_Z5AkI will render this:

Boxes / Callouts

Typing > Warning: There will be no second warning! will render this:

Warning: There will be no second warning!

Typing > Important: Pay attention! It's important. will render this:

Important: Pay attention! It’s important.

Typing > Tip: This is my tip. will render this:

Tip: This is my tip.

Typing > Note: Take note of this. will render this:

Note: Take note of this.

Typing > Note: A doc link to [an example website: fast.ai](https://www.fast.ai/) should also work fine. will render in the docs:

Note: A doc link to an example website: fast.ai should also work fine.

Footnotes

You can have footnotes in notebooks, however the syntax is different compared to markdown documents. This guide provides more detail about this syntax, which looks like this:

For example, here is a footnote {% fn 1 %}.
And another {% fn 2 %}
{{ 'This is the footnote.' | fndetail: 1 }}
{{ 'This is the other footnote. You can even have a [link](www.github.com)!' | fndetail: 2 }}

For example, here is a footnote ¹.

And another ²

1. This is the footnote.↩

2. This is the other footnote. You can even have a link!↩

	Title	Worldwide_Gross	Production_Budget	Distributor	MPAA_Rating	IMDB_Rating	Rotten_Tomatoes_Rating
0	The Land Girls	146083.0	8000000.0	Gramercy	R	6.1	NaN
1	First Love, Last Rites	10876.0	300000.0	Strand	R	6.9	NaN
2	I Married a Strange Person	203134.0	250000.0	Lionsgate	None	6.8	NaN
3	Let's Talk About Sex	373615.0	300000.0	Fine Line	None	NaN	13.0
4	Slam	1087521.0	1000000.0	Trimark	R	3.4	62.0

Tenzin’s Blog

What is code

Cost of bad code

Pytest Book CHAPTER 2, Writing Test Functions

Expecting Exceptions

Marking Test Functions

Skipping Tests

Marking Tests as Expecting to Fail

Parametrized Testing

Summary of SOLID Principles

What is SOLID Design

Single Responsibility Principle (SRP)

Open Closed Principle (OCP)

Liskov’s Substituitability Principle (LSP)

Interface Segregation Principle (ISP)

Dependency Inversion Principle (DIP)

References

A Summary of “Molecule of More”

Summary of “Molecule of More”

བརྩེ་བ།

Introduction to Unicode in Python

What is Unicode

UTF-8 - Great Hack of Unicode

UTF-8 Character Representation

Calulating Numeral value of a Unicode code point in python

Fun facts

Fastpages Notebook Blog Post

About

Front Matter

Markdown Shortcuts

Interactive Charts With Altair

Example 1: DropDown

Example 2: Tooltips

Example 3: More Tooltips

Data Tables

Images

Local Images

Remote Images

Animated Gifs

Captions

Other Elements

Tweetcards

Youtube Videos

Boxes / Callouts

Footnotes

An Example Markdown Post

Example Markdown Post

Basic setup

Basic formatting

Lists

Boxes and stuff

Images

Code

Tables

Tweetcards

Footnotes