Language Selection

English French German Italian Portuguese Spanish

OpenSource.com

Syndicate content
Updated: 2 hours 31 min ago

Drop your database for PostgreSQL

Saturday 24th of September 2022 07:00:00 AM
Drop your database for PostgreSQL Seth Kenlon Sat, 09/24/2022 - 03:00

Databases are tools to store information in an organized but flexible way. A spreadsheet is essentially a database, but the constraints of a graphical application render most spreadsheet applications useless to programmers. With Edge and IoT devices becoming significant target platforms, developers need powerful but lightweight solutions for storing, processing, and querying large amounts of data. One of my favourite combinations is the PostgreSQL database and Lua bindings, but the possibilities are endless. Whatever language you use, Postgres is a great choice for a database, but you need to know some basics before adopting it.

Install Postgres

To install PostgreSQL on Linux, use your software repository. On Fedora, CentOS, Mageia, and similar:

$ sudo dnf install postgresql postgresql-server

On Debian, Linux Mint, Elementary, and similar:

$ sudo apt install postgresql postgresql-contrib

On macOS and Windows, download an installer from postgresql.org.

Setting up Postgres

Most distributions install the Postgres database without starting it, but provide you with a script or systemd service to help it start reliably. However, before you start PostgreSQL, you must create a database cluster.

Fedora

On Fedora, CentOS, or similar, there's a Postgres setup script provided in the Postgres package. Run this script for easy configuration:

$ sudo /usr/bin/postgresql-setup --initdb
[sudo] password:
 * Initializing database in '/var/lib/pgsql/data'
 * Initialized, logs are in /var/lib/pgsql/initdb_postgresql.logDebian

On Debian-based distributions, setup is performed automatically by apt during installation.

Everything else

Finally, if you're running something else, then you can just use the toolchain provided by Postgres itself. The initdb command creates a database cluster, but you must run it as the postgres user, an identity you may temporarily assume using sudo:

$ sudo -u postgres \
"initdb -D /var/lib/pgsql/data \
--locale en_US.UTF-8 --auth md5 --pwprompt"Start Postgres

Now that a cluster exists, start the Postgres server using either the command provided to you in the output of initdb or with systemd:

$ sudo systemctl start postgresqlCreating a database user

To create a Postgres user, use the createuser command. The postgres user is the superuser of the Postgres install,

$ sudo -u postgres createuser --interactive --password bogus
Shall the new role be a superuser? (y/n) n
Shall the new role be allowed to create databases? (y/n) y
Shall the new role be allowed to create more new roles? (y/n) n
Password:Create a database

To create a new database, use the createdb command. In this example, I create the database exampledb and assign ownership of it to the user bogus:

$ createdb exampledb --owner bogusInteracting with PostgreSQL

You can interact with a PostgreSQL database using the psql command. This command provides an interactive shell so you can view and update your databases. To connect to a database, specify the user and database you want to use:

$ psql --user bogus exampledb
psql (XX.Y)
Type "help" for help.

exampledb=>Create a table

Databases contain tables, which can be visualized as a spreadsheet. There's a series of rows (called records in a database) and columns. The intersection of a row and a column is called a field.

The Structured Query Language (SQL) is named after what it provides: A method to inquire about the contents of a database in a predictable and consistent syntax to receive useful results.

Currently, your database is empty, devoid of any tables. You can create a table with the CREATE query. It's useful to combine this with the IF NOT EXISTS statement, which prevents PostgreSQL from clobbering an existing table.

Before you createa table, think about what kind of data (the "data type" in SQL terminology) you anticipate the table to contain. In this example, I create a table with one column for a unique identifier and one column for some arbitrary text up to nine characters.

exampledb=> CREATE TABLE IF NOT EXISTS my_sample_table(
exampledb(> id SERIAL,
exampledb(> wordlist VARCHAR(9) NOT NULL
);

The SERIAL keyword isn't actually a data type. It's special notation in PostgreSQL that creates an auto-incrementing integer field. The VARCHAR keyword is a data type indicating a variable number of characters within a limit. In this code, I've specified a maximum of 9 characters. There are lots of data types in PostgreSQL, so refer to the project documentation for a list of options.

Insert data

You can populate your new table with some sample data by using the INSERT SQL keyword:

exampledb=> INSERT INTO my_sample_table (wordlist) VALUES ('Alice');
INSERT 0 1

Your data entry fails, should you attempt to put more than 9 characters into the wordlist field:

exampledb=> INSERT INTO my_sample_table (WORDLIST) VALUES ('Alexandria');
ERROR:  VALUE too long FOR TYPE CHARACTER VARYING(9)Alter a table or column

When you need to change a field definition, you use the ALTER SQL keyword. For instance, should you decide that a nine character limit for wordlist, you can increase its allowance by setting its data type:

exampledb=> ALTER TABLE my_sample_table
ALTER COLUMN wordlist SET DATA TYPE VARCHAR(10);
ALTER TABLE
exampledb=> INSERT INTO my_sample_table (WORDLIST) VALUES ('Alexandria');
INSERT 0 1View data in a table

SQL is a query language, so you view the contents of a database through queries. Queries can be simple, or it can involve joining complex relationships between several different tables. To see everything in a table, use the SELECT keyword on * (an asterisk is a wildcard):

exampledb=> SELECT * FROM my_sample_table;
 id |  wordlist
\----+------------
  1 | Alice
  2 | Bob
  3 | Alexandria
(3 ROWS)More data

PostgreSQL can handle a lot of data, but as with any database the key to success is how you design your database for storage and what you do with the data once you've got it stored. A relatively large public data set can be found on OECD.org, and using this you can try some advanced database techniques.

First, download the data as comma-separated values (CSV) and save the file as land-cover.csv in your Downloads folder.

Browse the data in a text editor or spreadsheet application to get an idea of what columns there are, and what kind of data each column contains. Look at the data carefully and keep an eye out for exceptions to an apparent rule. For instance, the COU column, containing a country code such as AUS for Australia and GRC for Greece, tends to be 3 characters until the oddity BRIICS.

Once you understand the data you're working with, you can prepare a Postgres database:

$ createdb landcoverdb --owner bogus
$ psql --user bogus landcoverdb
landcoverdb=> create table land_cover(
country_code varchar(6),
country_name varchar(76),
small_subnational_region_code varchar(5),
small_subnational_region_name varchar(14),
large_subnational_region_code varchar(17),
large_subnational_region_name varchar(44),
measure_code varchar(13),
measure_name varchar(29),
land_cover_class_code varchar(17),
land_cover_class_name varchar(19),
year_code integer,
year_value integer,
unit_code varchar(3),
unit_name varchar(17),
power_code integer,
power_name varchar(9),
reference_period_code varchar(1),
reference_period_name varchar(1),
value float(8),
flag_codes varchar(1),
flag_names varchar(1));Importing data

Postgres can import CSV data directly using the special metacommand \copy:

landcoverdb=> \copy land_cover from '~/land-cover.csv' with csv header delimiter ','
COPY 22113

That's 22,113 records imported. Seems like a good start!

More on edge computing Understanding edge computing Why Linux is critical to edge computing eBook: Running Kubernetes on your Raspberry Pi Download now: The automated enterprise eBook eBook: A practical guide to home automation using open source tools eBook: 7 examples of automation on the edge The latest on edge Querying data

A broad SELECT statement to see all columns of all 22,113 records is possible, and Postgres very nicely pipes the output to a screen pager so you can scroll through the output at a leisurely pace. However, using advanced SQL you can get some useful views of what's otherwise some pretty raw data.

landcoverdb=> SELECT
    lcm.country_name,
    lcm.year_value,
    SUM(lcm.value) sum_value
FROM land_cover lcm
JOIN (
    SELECT
        country_name,
        large_subnational_region_name,
        small_subnational_region_name,
        MAX(year_value) max_year_value
    FROM land_cover
    GROUP BY country_name,
        large_subnational_region_name,
        small_subnational_region_name
) AS lcmyv
ON
    lcm.country_name = lcmyv.country_name AND
    lcm.large_subnational_region_name = lcmyv.large_subnational_region_name AND
    lcm.small_subnational_region_name = lcmyv.small_subnational_region_name AND
    lcm.year_value = lcmyv.max_year_value
GROUP BY lcm.country_name,
    lcm.large_subnational_region_name,
    lcm.small_subnational_region_name,
    lcm.year_value
ORDER BY country_name,
    year_value;

Here's some sample output:

\---------------+------------+------------
 Afghanistan    |       2019 |  743.48425
 Albania        |       2019 |  128.82532
 Algeria        |       2019 |  2417.3281
 American Samoa |       2019 |   100.2007
 Andorra        |       2019 |  100.45613
 Angola         |       2019 |  1354.2192
 Anguilla       |       2019 | 100.078514
 Antarctica     |       2019 |  12561.907
[...]

SQL is a rich langauge, and so it's beyond the scope of this article. Read through the SQL code and see if you can modify it to provide a different set of data.

Open database

PostgreSQL is one of the great open source databases. With it, you can design repositories for structured data, and then use SQL to view it in different ways so you can gain fresh perspectives on that data. Postgres integrates with many languages, including Python, Lua, Groovy, Java, and more, so regardless of your toolset, you can probably make use of this excellent database.

Postgres is one of the most flexible databases available, and it's open source.

Image by:

Image by Mapbox Uncharted ERG, CC-BY 3.0 US

Databases Edge computing What to read next Install MariaDB or MySQL on Linux 3 ways to use PostgreSQL commands This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. 6870 points (Correspondent) Vancouver, Canada

Seldom without a computer of some sort since graduating from the University of British Columbia in 1978, I have been a full-time Linux user since 2005, a full-time Solaris and SunOS user from 1986 through 2005, and UNIX System V user before that.

On the technical side of things, I have spent a great deal of my career as a consultant, doing data analysis and visualization; especially spatial data analysis. I have a substantial amount of related programming experience, using C, awk, Java, Python, PostgreSQL, PostGIS and lately Groovy. I'm looking at Julia with great interest. I have also built a few desktop and web-based applications, primarily in Java and lately in Grails with lots of JavaScript on the front end and PostgreSQL as my database of choice.

Aside from that, I spend a considerable amount of time writing proposals, technical reports and - of course - stuff on https://www.opensource.com.

User Attributes Correspondent Open Sourcerer People's Choice Award 100+ Contributions Club Emerging Contributor Award 2016 Contributor Club Author Comment Gardener Correspondent Columnist 2 Comments Register or Login to post a comment. Greg Pittman | September 24, 2022 Register or Login to like

After you do
sudo systemctl start postgresql.service
you can do
sudo systemctl enable postgresql.service
so that the postgresql server starts each time you boot your machine.
Incidentally, my system needs the .service after postgresql.

Heiko Staudt | September 25, 2022 Register or Login to like

I use postgreSQL for some years now.
I switched because it has own data types for IPv4 and IPv6 addresses like inet and cidr . So i can query a table for IPs which are in a given network. Oh, and not to forget the macaddr.
My personal "dream-team" is postgreSQL with Flask or Django.

How to build a dynamic distributed database with DistSQL

Friday 23rd of September 2022 07:00:00 AM
How to build a dynamic distributed database with DistSQL Raigor Jiang Fri, 09/23/2022 - 03:00

Distributed databases are common for many reasons. They increase reliability, redundancy, and performance. Apache ShardingSphere is an open source framework that enables you to transform any database into a distributed database. Since the release of ShardingSphere 5.0.0, DistSQL (Distributed SQL) has provided dynamic management for the ShardingSphere ecosystem.

In this article, I demonstrate a data sharding scenario in which DistSQL's flexibility allows you to create a distributed database. At the same time, I show some syntax sugar to simplify operating procedures, allowing your potential users to choose their preferred syntax.

A series of DistSQL statements are run through practical cases to give you a complete set of practical DistSQL sharding management methods, which create and maintain distributed databases through dynamic management.

Image by:

(Jiang Longtao, CC BY-SA 4.0)

What is sharding?

In database terminology, sharding is the process of partitioning a table into separate entities. While the table data is directly related, it often exists on different physical database nodes or, at the very least, within separate logical partitions.

Practical case example

To follow along with this example, you must have these components in place, either in your lab or in your mind as you read this article:

  • Two sharding tables: t_order and t_order_item.
  • For both tables, database shards are carried out with the user_id field, and table shards with the order_id field.
  • The number of shards is two databases times three tables.
Image by:

(Jiang Longtao, CC BY-SA 4.0)

Set up the environment

1. Prepare a database (MySQL, MariaDB, PostgreSQL, or openGauss) instance for access. Create two new databases: demo_ds_0 and demo_ds_1.

2. Deploy Apache ShardingSphere-Proxy 5.1.2 and Apache ZooKeeper. ZooKeeper acts as a governance center and stores ShardingSphere metadata information.

3. Configure server.yaml in the Proxy conf directory as follows:

mode:
  type: Cluster
  repository:
    type: ZooKeeper
    props:
      namespace: governance_ds
      server-lists: localhost:2181 #ZooKeeper address
      retryIntervalMilliseconds: 500
      timeToLiveSeconds: 60
      maxRetries: 3
      operationTimeoutMilliseconds: 500
  overwrite: falserules:
 - !AUTHORITY
    users:
     - root@%:root

4. Start ShardingSphere-Proxy and connect it to Proxy using a client, for example:

$ mysql -h 127.0.0.1 -P 3307 -u root -p

5. Create a distributed database:

CREATE DATABASE sharding_db;USE sharding_db;Add storage resources

Next, add storage resources corresponding to the database:

ADD RESOURCE ds_0 (
    HOST=127.0.0.1,
    PORT=3306,
    DB=demo_ds_0,
    USER=root,
    PASSWORD=123456
), ds_1(
    HOST=127.0.0.1,
    PORT=3306,
    DB=demo_ds_1,
    USER=root,
    PASSWORD=123456
);

View the storage resources:

mysql> SHOW DATABASE RESOURCES\G;
******** 1. row ***************************
         name: ds_1
         type: MySQL
         host: 127.0.0.1
         port: 3306
           db: demo_ds_1
          -- Omit partial attributes
******** 2. row ***************************
         name: ds_0
         type: MySQL
         host: 127.0.0.1
         port: 3306
           db: demo_ds_0
          -- Omit partial attributes

Adding the optional \G switch to the query statement makes the output format easy to read.

More on data science What is data science? What is Python? How to become a data scientist Data scientist: A day in the life What is big data? Whitepaper: Data-intensive intelligent applications in a hybrid cloud blueprint MariaDB and MySQL cheat sheet Latest data science articles Create sharding rules

ShardingSphere's sharding rules support regular sharding and automatic sharding. Both sharding methods have the same effect. The difference is that the configuration of automatic sharding is more concise, while regular sharding is more flexible and independent.

Refer to the following links for more details on automatic sharding:

Next, it's time to adopt regular sharding and use the INLINE expression algorithm to implement the sharding scenarios described in the requirements.

Primary key generator

The primary key generator creates a secure and unique primary key for a data table in a distributed scenario. For details, refer to the document Distributed Primary Key.

1. Create a primary key generator:

CREATE SHARDING KEY GENERATOR snowflake_key_generator (
TYPE(NAME=SNOWFLAKE)
);

2. Query the primary key generator:

mysql> SHOW SHARDING KEY GENERATORS;
+-------------------------+-----------+-------+
| name                    | type      | props |
+-------------------------+-----------+-------+
| snowflake_key_generator | snowflake | {}    |
+-------------------------+-----------+-------+
1 row in set (0.01 sec)Sharding algorithm

1. Create a database sharding algorithm used by t_order and t_order_item in common:

-- Modulo 2 based on user_id in database sharding
CREATE SHARDING ALGORITHM database_inline (
TYPE(NAME=INLINE,PROPERTIES("algorithm-expression"="ds_${user_id % 2}"))
);

2. Create different table shards algorithms for t_order and t_order_item:

-- Modulo 3 based on order_id in table sharding
CREATE SHARDING ALGORITHM t_order_inline (
TYPE(NAME=INLINE,PROPERTIES("algorithm-expression"="t_order_${order_id % 3}"))
);
CREATE SHARDING ALGORITHM t_order_item_inline (
TYPE(NAME=INLINE,PROPERTIES("algorithm-expression"="t_order_item_${order_id % 3}"))
);

3. Query the sharding algorithm:

mysql> SHOW SHARDING ALGORITHMS;
+---------------------+--------+---------------------------------------------------+
| name                | type   | props                                             |
+---------------------+--------+---------------------------------------------------+
| database_inline     | inline | algorithm-expression=ds_${user_id % 2}            |
| t_order_inline      | inline | algorithm-expression=t_order_${order_id % 3}      |
| t_order_item_inline | inline | algorithm-expression=t_order_item_${order_id % 3} |
+---------------------+--------+---------------------------------------------------+
3 rows in set (0.00 sec)Create a default sharding strategy

The sharding strategy consists of a sharding key and sharding algorithm, which in this case is databaseStrategy and tableStrategy. Because t_order and t_order_item have the same database sharding field and sharding algorithm, create a default strategy to be used by all shard tables with no sharding strategy configured.

1. Create a default database sharding strategy:

CREATE DEFAULT SHARDING DATABASE STRATEGY (
TYPE=STANDARD,SHARDING_COLUMN=user_id,SHARDING_ALGORITHM=database_inline
);

2. Query default strategy:

mysql> SHOW DEFAULT SHARDING STRATEGY\G;
*************************** 1. row ***************************
                    name: TABLE
                    type: NONE
         sharding_column:
 sharding_algorithm_name:
 sharding_algorithm_type:
sharding_algorithm_props:
*************************** 2. row ***************************
                    name: DATABASE
                    type: STANDARD
         sharding_column: user_id
 sharding_algorithm_name: database_inline
 sharding_algorithm_type: inline
sharding_algorithm_props: {algorithm-expression=ds_${user_id % 2}}
2 rows in set (0.00 sec)

You have not configured the default table sharding strategy, so the default strategy of TABLE is NONE.

Set sharding rules

The primary key generator and sharding algorithm are both ready. Now you can create sharding rules. The method I demonstrate below is a little complicated and involves multiple steps. In the next section, I'll show you how to create sharding rules in just one step, but for now, witness how it's typically done.

First, define t_order:

CREATE SHARDING TABLE RULE t_order (
DATANODES("ds_${0..1}.t_order_${0..2}"),
TABLE_STRATEGY(TYPE=STANDARD,SHARDING_COLUMN=order_id,SHARDING_ALGORITHM=t_order_inline),
KEY_GENERATE_STRATEGY(COLUMN=order_id,KEY_GENERATOR=snowflake_key_generator)
);

Here is an explanation of the values found above:

  • DATANODES specifies the data nodes of shard tables.
  • TABLE_STRATEGY specifies the table strategy, among which SHARDING_ALGORITHM uses created sharding algorithm t_order_inline.
  • KEY_GENERATE_STRATEGY specifies the primary key generation strategy of the table. Skip this configuration if primary key generation is not required.

Next, define t_order_item:

CREATE SHARDING TABLE RULE t_order_item (
DATANODES("ds_${0..1}.t_order_item_${0..2}"),
TABLE_STRATEGY(TYPE=STANDARD,SHARDING_COLUMN=order_id,SHARDING_ALGORITHM=t_order_item_inline),
KEY_GENERATE_STRATEGY(COLUMN=order_item_id,KEY_GENERATOR=snowflake_key_generator)
);

Query the sharding rules to verify what you've created:

mysql> SHOW SHARDING TABLE RULES\G;
************************** 1. row ***************************
                           table: t_order
               actual_data_nodes: ds_${0..1}.t_order_${0..2}
             actual_data_sources:
          database_strategy_type: STANDARD
        database_sharding_column: user_id
database_sharding_algorithm_type: inline
database_sharding_algorithm_props: algorithm-expression=ds_${user_id % 2}
              table_strategy_type: STANDARD
            table_sharding_column: order_id
    table_sharding_algorithm_type: inline
   table_sharding_algorithm_props: algorithm-expression=t_order_${order_id % 3}
              key_generate_column: order_id
               key_generator_type: snowflake
              key_generator_props:
*************************** 2. row ***************************
                            table: t_order_item
                actual_data_nodes: ds_${0..1}.t_order_item_${0..2}
              actual_data_sources:
           database_strategy_type: STANDARD
         database_sharding_column: user_id
 database_sharding_algorithm_type: inline
database_sharding_algorithm_props: algorithm-expression=ds_${user_id % 2}
              table_strategy_type: STANDARD
            table_sharding_column: order_id
    table_sharding_algorithm_type: inline
   table_sharding_algorithm_props: algorithm-expression=t_order_item_${order_id % 3}
              key_generate_column: order_item_id
               key_generator_type: snowflake
              key_generator_props:
2 rows in set (0.00 sec)

This looks right so far. You have now configured the sharding rules for t_order and t_order_item.

You can skip the steps for creating the primary key generator, sharding algorithm, and default strategy, and complete the sharding rules in one step. Here's how to make it easier.

Sharding rule syntax

For instance, if you want to add a shard table called t_order_detail, you can create sharding rules as follows:

CREATE SHARDING TABLE RULE t_order_detail (
DATANODES("ds_${0..1}.t_order_detail_${0..1}"),
DATABASE_STRATEGY(TYPE=STANDARD,SHARDING_COLUMN=user_id,SHARDING_ALGORITHM(TYPE(NAME=INLINE,PROPERTIES("algorithm-expression"="ds_${user_id % 2}")))),
TABLE_STRATEGY(TYPE=STANDARD,SHARDING_COLUMN=order_id,SHARDING_ALGORITHM(TYPE(NAME=INLINE,PROPERTIES("algorithm-expression"="t_order_detail_${order_id % 3}")))),
KEY_GENERATE_STRATEGY(COLUMN=detail_id,TYPE(NAME=snowflake))
);

This statement specifies a database sharding strategy, table strategy, and primary key generation strategy, but it doesn't use existing algorithms. The DistSQL engine automatically uses the input expression to create an algorithm for the sharding rules of t_order_detail.

Now there's a primary key generator:

mysql> SHOW SHARDING KEY GENERATORS;
+--------------------------+-----------+-------+
| name                     | type      | props |
+--------------------------+-----------+-------+
| snowflake_key_generator  | snowflake | {}    |
| t_order_detail_snowflake | snowflake | {}    |
+--------------------------+-----------+-------+
2 rows in set (0.00 sec)

Display the sharding algorithm:

mysql> SHOW SHARDING ALGORITHMS;
+--------------------------------+--------+-----------------------------------------------------+
| name                           | type   | props                                               |
+--------------------------------+--------+-----------------------------------------------------+
| database_inline                | inline | algorithm-expression=ds_${user_id % 2}              |
| t_order_inline                 | inline | algorithm-expression=t_order_${order_id % 3}        |
| t_order_item_inline            | inline | algorithm-expression=t_order_item_${order_id % 3}   |
| t_order_detail_database_inline | inline | algorithm-expression=ds_${user_id % 2}              |
| t_order_detail_table_inline    | inline | algorithm-expression=t_order_detail_${order_id % 3} |
+--------------------------------+--------+-----------------------------------------------------+
5 rows in set (0.00 sec)

And finally, the sharding rules:

mysql> SHOW SHARDING TABLE RULES\G;
*************************** 1. row ***************************
                            table: t_order
                actual_data_nodes: ds_${0..1}.t_order_${0..2}
              actual_data_sources:
           database_strategy_type: STANDARD
         database_sharding_column: user_id
 database_sharding_algorithm_type: inline
database_sharding_algorithm_props: algorithm-expression=ds_${user_id % 2}
              table_strategy_type: STANDARD
            table_sharding_column: order_id
    table_sharding_algorithm_type: inline
   table_sharding_algorithm_props: algorithm-expression=t_order_${order_id % 3}
              key_generate_column: order_id
               key_generator_type: snowflake
              key_generator_props:
*************************** 2. row ***************************
                            table: t_order_item
                actual_data_nodes: ds_${0..1}.t_order_item_${0..2}
              actual_data_sources:
           database_strategy_type: STANDARD
         database_sharding_column: user_id
 database_sharding_algorithm_type: inline
database_sharding_algorithm_props: algorithm-expression=ds_${user_id % 2}
              table_strategy_type: STANDARD
            table_sharding_column: order_id
    table_sharding_algorithm_type: inline
   table_sharding_algorithm_props: algorithm-expression=t_order_item_${order_id % 3}
              key_generate_column: order_item_id
               key_generator_type: snowflake
              key_generator_props:
*************************** 3. row ***************************
                            table: t_order_detail
                actual_data_nodes: ds_${0..1}.t_order_detail_${0..1}
              actual_data_sources:
           database_strategy_type: STANDARD
         database_sharding_column: user_id
 database_sharding_algorithm_type: inline
database_sharding_algorithm_props: algorithm-expression=ds_${user_id % 2}
              table_strategy_type: STANDARD
            table_sharding_column: order_id
    table_sharding_algorithm_type: inline
   table_sharding_algorithm_props: algorithm-expression=t_order_detail_${order_id % 3}
              key_generate_column: detail_id
               key_generator_type: snowflake
              key_generator_props:
3 rows in set (0.01 sec)

In the CREATE SHARDING TABLE RULE statement, DATABASE_STRATEGY, TABLE_STRATEGY, and KEY_GENERATE_STRATEGY can reuse existing algorithms.

Alternatively, they can be defined quickly through syntax. The difference is that additional algorithm objects are created.

Configuration and verification

Once you have created the configuration verification rules, you can verify them in the following ways.

1. Check node distribution:

DistSQL provides SHOW SHARDING TABLE NODES for checking node distribution, and users can quickly learn the distribution of shard tables:

mysql> SHOW SHARDING TABLE NODES;
+----------------+------------------------------------------------------------------------------------------------------------------------------+
| name           | nodes                                                                                                                        |
+----------------+------------------------------------------------------------------------------------------------------------------------------+
| t_order        | ds_0.t_order_0, ds_0.t_order_1, ds_0.t_order_2, ds_1.t_order_0, ds_1.t_order_1, ds_1.t_order_2                               |
| t_order_item   | ds_0.t_order_item_0, ds_0.t_order_item_1, ds_0.t_order_item_2, ds_1.t_order_item_0, ds_1.t_order_item_1, ds_1.t_order_item_2 |
| t_order_detail | ds_0.t_order_detail_0, ds_0.t_order_detail_1, ds_1.t_order_detail_0, ds_1.t_order_detail_1                                   |
+----------------+------------------------------------------------------------------------------------------------------------------------------+
3 rows in set (0.01 sec)

mysql> SHOW SHARDING TABLE NODES t_order_item;
+--------------+------------------------------------------------------------------------------------------------------------------------------+
| name         | nodes                                                                                                                        |
+--------------+------------------------------------------------------------------------------------------------------------------------------+
| t_order_item | ds_0.t_order_item_0, ds_0.t_order_item_1, ds_0.t_order_item_2, ds_1.t_order_item_0, ds_1.t_order_item_1, ds_1.t_order_item_2 |
+--------------+------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

You can see that the node distribution of the shard table is consistent with what is described in the requirement.

SQL preview

Previewing SQL is also an easy way to verify configurations. Its syntax is PREVIEW SQL. First, make a query with no shard key, with all routes:

mysql> PREVIEW SELECT * FROM t_order;
+------------------+---------------------------------------------------------------------------------------------+
| data_source_name | actual_sql                                                                                  |
+------------------+---------------------------------------------------------------------------------------------+
| ds_0             | SELECT * FROM t_order_0 UNION ALL SELECT * FROM t_order_1 UNION ALL SELECT * FROM t_order_2 |
| ds_1             | SELECT * FROM t_order_0 UNION ALL SELECT * FROM t_order_1 UNION ALL SELECT * FROM t_order_2 |
+------------------+---------------------------------------------------------------------------------------------+
2 rows in set (0.13 sec)

mysql> PREVIEW SELECT * FROM t_order_item;
+------------------+------------------------------------------------------------------------------------------------------------+
| data_source_name | actual_sql                                                                                                 |
+------------------+------------------------------------------------------------------------------------------------------------+
| ds_0             | SELECT * FROM t_order_item_0 UNION ALL SELECT * FROM t_order_item_1 UNION ALL SELECT * FROM t_order_item_2 |
| ds_1             | SELECT * FROM t_order_item_0 UNION ALL SELECT * FROM t_order_item_1 UNION ALL SELECT * FROM t_order_item_2 |
+------------------+------------------------------------------------------------------------------------------------------------+
2 rows in set (0.00 sec)

Now specify user_id in a query with a single database route:

mysql> PREVIEW SELECT * FROM t_order WHERE user_id = 1;
+------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+
| data_source_name | actual_sql                                                                                                                                        |
+------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+
| ds_1             | SELECT * FROM t_order_0 WHERE user_id = 1 UNION ALL SELECT * FROM t_order_1 WHERE user_id = 1 UNION ALL SELECT * FROM t_order_2 WHERE user_id = 1 |
+------------------+---------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.14 sec)

mysql> PREVIEW SELECT * FROM t_order_item WHERE user_id = 2;
+------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| data_source_name | actual_sql                                                                                                                                                       |
+------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| ds_0             | SELECT * FROM t_order_item_0 WHERE user_id = 2 UNION ALL SELECT * FROM t_order_item_1 WHERE user_id = 2 UNION ALL SELECT * FROM t_order_item_2 WHERE user_id = 2 |
+------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

Specify user_id and order_id with a single table route:

mysql> PREVIEW SELECT * FROM t_order WHERE user_id = 1 AND order_id = 1;
+------------------+------------------------------------------------------------+
| data_source_name | actual_sql                                                 |
+------------------+------------------------------------------------------------+
| ds_1             | SELECT * FROM t_order_1 WHERE user_id = 1 AND order_id = 1 |
+------------------+------------------------------------------------------------+
1 row in set (0.04 sec)

mysql> PREVIEW SELECT * FROM t_order_item WHERE user_id = 2 AND order_id = 5;
+------------------+-----------------------------------------------------------------+
| data_source_name | actual_sql                                                      |
+------------------+-----------------------------------------------------------------+
| ds_0             | SELECT * FROM t_order_item_2 WHERE user_id = 2 AND order_id = 5 |
+------------------+-----------------------------------------------------------------+
1 row in set (0.01 sec)

Single-table routes scan the fewest shard tables and offer the highest efficiency.

Query unused resources

During system maintenance, algorithms or storage resources that are no longer in use may need to be released, or resources that need to be released may have been referenced and cannot be deleted. DistSQL's SHOW UNUSED RESOURCES command can solve these problems:

mysql> ADD RESOURCE ds_2 (
    ->     HOST=127.0.0.1,
    ->     PORT=3306,
    ->     DB=demo_ds_2,
    ->     USER=root,
    ->     PASSWORD=123456
    -> );
Query OK, 0 rows affected (0.07 sec)

mysql> SHOW UNUSED RESOURCES\G;
*************************** 1. row ***************************
                           name: ds_2
                           type: MySQL
                           host: 127.0.0.1
                           port: 3306
                             db: demo_ds_2
connection_timeout_milliseconds: 30000
      idle_timeout_milliseconds: 60000
      max_lifetime_milliseconds: 2100000
                  max_pool_size: 50
                  min_pool_size: 1
                      read_only: false
               other_attributes: {"dataSourceProperties":{"cacheServerConfiguration":"true","elideSetAutoCommits":"true","useServerPrepStmts":"true","cachePrepStmts":"true","useSSL":"false","rewriteBatchedStatements":"true","cacheResultSetMetadata":"false","useLocalSessionState":"true","maintainTimeStats":"false","prepStmtCacheSize":"200000","tinyInt1isBit":"false","prepStmtCacheSqlLimit":"2048","serverTimezone":"UTC","netTimeoutForStreamingResults":"0","zeroDateTimeBehavior":"round"},"healthCheckProperties":{},"initializationFailTimeout":1,"validationTimeout":5000,"leakDetectionThreshold":0,"poolName":"HikariPool-8","registerMbeans":false,"allowPoolSuspension":false,"autoCommit":true,"isolateInternalQueries":false}
1 row in set (0.03 sec)Query unused primary key generator

DistSQL can also display unused sharding key generators with the SHOW UNUSED SHARDING KEY GENERATORS:

mysql> SHOW SHARDING KEY GENERATORS;
+--------------------------+-----------+-------+
| name                     | type      | props |
+--------------------------+-----------+-------+
| snowflake_key_generator  | snowflake | {}    |
| t_order_detail_snowflake | snowflake | {}    |
+--------------------------+-----------+-------+
2 rows in set (0.00 sec)

mysql> SHOW UNUSED SHARDING KEY GENERATORS;
Empty set (0.01 sec)

mysql> CREATE SHARDING KEY GENERATOR useless (
    -> TYPE(NAME=SNOWFLAKE)
    -> );
Query OK, 0 rows affected (0.04 sec)

mysql> SHOW UNUSED SHARDING KEY GENERATORS;
+---------+-----------+-------+
| name    | type      | props |
+---------+-----------+-------+
| useless | snowflake |       |
+---------+-----------+-------+
1 row in set (0.01 sec)Query unused sharding algorithm

DistSQL can reveal unused sharding algorithms with (you guessed it) the SHOW UNUSED SHARDING ALGORITHMS command:

mysql> SHOW SHARDING ALGORITHMS;
+--------------------------------+--------+-----------------------------------------------------+
| name                           | type   | props                                               |
+--------------------------------+--------+-----------------------------------------------------+
| database_inline                | inline | algorithm-expression=ds_${user_id % 2}              |
| t_order_inline                 | inline | algorithm-expression=t_order_${order_id % 3}        |
| t_order_item_inline            | inline | algorithm-expression=t_order_item_${order_id % 3}   |
| t_order_detail_database_inline | inline | algorithm-expression=ds_${user_id % 2}              |
| t_order_detail_table_inline    | inline | algorithm-expression=t_order_detail_${order_id % 3} |
+--------------------------------+--------+-----------------------------------------------------+
5 rows in set (0.00 sec)

mysql> CREATE SHARDING ALGORITHM useless (
    -> TYPE(NAME=INLINE,PROPERTIES("algorithm-expression"="ds_${user_id % 2}"))
    -> );
Query OK, 0 rows affected (0.04 sec)

mysql> SHOW UNUSED SHARDING ALGORITHMS;
+---------+--------+----------------------------------------+
| name    | type   | props                                  |
+---------+--------+----------------------------------------+
| useless | inline | algorithm-expression=ds_${user_id % 2} |
+---------+--------+----------------------------------------+
1 row in set (0.00 sec)Query rules that use the target storage resources

You can also see used resources within rules with SHOW RULES USED RESOURCE. All rules that use a resource can be queried, not limited to the sharding rule.

mysql> DROP RESOURCE ds_0;
ERROR 1101 (C1101): Resource [ds_0] is still used by [ShardingRule].

mysql> SHOW RULES USED RESOURCE ds_0;
+----------+----------------+
| type     | name           |
+----------+----------------+
| sharding | t_order        |
| sharding | t_order_item   |
| sharding | t_order_detail |
+----------+----------------+
3 rows in set (0.00 sec)Query sharding rules that use the target primary key generator

You can find sharding rules using a key generator with SHOW SHARDING TABLE RULES USED KEY GENERATOR:

mysql> SHOW SHARDING KEY GENERATORS;
+--------------------------+-----------+-------+
| name                     | type      | props |
+--------------------------+-----------+-------+
| snowflake_key_generator  | snowflake | {}    |
| t_order_detail_snowflake | snowflake | {}    |
| useless                  | snowflake | {}    |
+--------------------------+-----------+-------+
3 rows in set (0.00 sec)

mysql> DROP SHARDING KEY GENERATOR snowflake_key_generator;
ERROR 1121 (C1121): Sharding key generator `[snowflake_key_generator]` in database `sharding_db` are still in used.

mysql> SHOW SHARDING TABLE RULES USED KEY GENERATOR snowflake_key_generator;
+-------+--------------+
| type  | name         |
+-------+--------------+
| table | t_order      |
| table | t_order_item |
+-------+--------------+
2 rows in set (0.00 sec)Query sharding rules that use the target algorithm

Show sharding rules using a target algorithm with SHOW SHARDING TABLE RULES USED ALGORITHM:

mysql> SHOW SHARDING ALGORITHMS;
+--------------------------------+--------+-----------------------------------------------------+
| name                           | type   | props                                               |
+--------------------------------+--------+-----------------------------------------------------+
| database_inline                | inline | algorithm-expression=ds_${user_id % 2}              |
| t_order_inline                 | inline | algorithm-expression=t_order_${order_id % 3}        |
| t_order_item_inline            | inline | algorithm-expression=t_order_item_${order_id % 3}   |
| t_order_detail_database_inline | inline | algorithm-expression=ds_${user_id % 2}              |
| t_order_detail_table_inline    | inline | algorithm-expression=t_order_detail_${order_id % 3} |
| useless                        | inline | algorithm-expression=ds_${user_id % 2}              |
+--------------------------------+--------+-----------------------------------------------------+
6 rows in set (0.00 sec)

mysql> DROP SHARDING ALGORITHM t_order_detail_table_inline;
ERROR 1116 (C1116): Sharding algorithms `[t_order_detail_table_inline]` in database `sharding_db` are still in used.

mysql> SHOW SHARDING TABLE RULES USED ALGORITHM t_order_detail_table_inline;
+-------+----------------+
| type  | name           |
+-------+----------------+
| table | t_order_detail |
+-------+----------------+
1 row in set (0.00 sec)Make sharding better

DistSQL provides a flexible syntax to help simplify operations. In addition to the INLINE algorithm, DistSQL supports standard sharding, compound sharding, HINT sharding, and custom sharding algorithms.

If you have any questions or suggestions about Apache ShardingSphere, please feel free to post them on ShardingSphereGitHub.

Take a look at a data sharding scenario in which DistSQL's flexibility allows you to create a distributed database.

Image by:

Opensource.com

Databases What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

Install JDBC on Linux in 3 steps

Friday 23rd of September 2022 07:00:00 AM
Install JDBC on Linux in 3 steps Seth Kenlon Fri, 09/23/2022 - 03:00

When you write an application, it's common to require data storage. Sometimes you're storing assets your application needs to function, and other times you're storing user data, including preferences and save data. One way to store data is in a database, and in order to communicate between your code and a database, you need a database binding or connector for your language. For Java, a common database connector is JDBC (Java database connectivity.)

1. Install Java

Of course, to develop with Java you must also have Java installed. I recommend SDKman for Linux, macOS, and WSL or Cygwin. For Windows, you can download OpenJDK from developers.redhat.com.

More on Java What is enterprise Java programming? Red Hat build of OpenJDK Java cheat sheet Free online course: Developing cloud-native applications with microservices Fresh Java articles 2. Install JDBC with Maven

JDBC is an API, imported into your code with the statement import java.sql.*, but for it to be useful you must have a database driver and a database installed for it to interact with. The database driver you use and the database you want to communicate with must match: to interact with MySQL, you need a MySQL driver, to interact with SQLite3, you must have the SQLite3 driver, and so on.

For this article, I use PostgreSQL, but all the major databases, including MariaDB and SQLite3, have JDBC drivers.

You can download JDBC for PostgreSQL from jdbc.postgresql.org. I use Maven to manage Java dependencies, so I include it in pom.xml (adjusting the version number for what's current on Maven Central):

>
    >org.postgresql>
    >postgresql>
    >42.5.0>
>3. Install the database

You have to install the database you want to connect to through JDBC. There are several very good open source databases, but I had to choose one for this article, so I chose PostgreSQL.

To install PostgreSQL on Linux, use your software repository. On Fedora, CentOS, Mageia, and similar:

$ sudo dnf install postgresql postgresql-server

On Debian, Linux Mint, Elementary, and similar:

$ sudo apt install postgresql postgresql-contribDatabase connectivity

If you're not using PostgreSQL, the same general process applies:

  1. Install Java.

  2. Find the JDBC driver for your database of choice and include it in your pom.xml file.

  3. Install the database (server and client) on your development OS.

Three steps and you're ready to start writing code.

Install Java, install JDBC with Maven, and install the database. Then you're ready to interact with databases in your Java code.

Image by:

Pixabay. CC0.

Java Linux Databases What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

5 Git configurations I make on Linux

Thursday 22nd of September 2022 07:00:00 AM
5 Git configurations I make on Linux Alan Formy-Duval Thu, 09/22/2022 - 03:00

Setting up Git on Linux is simple, but here are the five things I do to get the perfect configuration:

  1. Create global configuration
  2. Set default name
  3. Set default email address
  4. Set default branch name
  5. Set default editor

I manage my code, shell scripts, and documentation versioning using Git. This means that for each new project I start, the first step is to create a directory for its content and make it into a Git repository:

$ mkdir newproject
$ cd newproject
$ git init

More on Git What is Git? Git cheat sheet Markdown cheat sheet New Git articles

There are certain general settings that I always want. Not many, but enough that I don't want to have to repeat the configuration each time. I like to take advantage of the global configuration capability of Git.

Git offers the git config command for manual configuration but this is a lot of work with certain caveats. For example, a common item to set is your email address. You can set it by running git config user.email followed by your email address. However, this will only take effect if you are in an existing Git directory:

$ git config user.email alan@opensource.com
fatal: not in a git directory

Plus, when this command is run within a Git repository, it only configures that specific one. The process must be repeated for new repositories. I can avoid this repetition by setting it globally. The --global option will instruct Git to write the email address to the global configuration file; ~/.gitconfig, even creating it if necessary:

  Remember, the tilde (~) character represents your home directory. In my case that is /home/alan. $ git config --global user.email alan@opensource.com
$ cat ~/.gitconfig
[user]
        email = alan@opensource.com

The downside here is if you have a large list of preferred settings, you will have a lot of commands to enter. This is time-consuming and prone to human error. Git provides an even more efficient and convenient way to directly edit your global configuration file—that is the first item on my list!

1. Create global configuration

If you have just started using Git, you may not have this file at all. Not to worry, let's skip the searching and get started. Just use the --edit option:

$ git config --global --edit

If no file is found, Git will generate one with the following content and open it in your shell environment's default editor:

# This is Git's per-user configuration file.
[user]
# Please adapt and uncomment the following lines:
#       name = Alan
#       email = alan@hopper
~
~
~
"~/.gitconfig" 5L, 155B                                     1,1           All

Now that we have opened the editor and Git has created the global configuration file behind the scenes, we can continue with the rest of the settings.

2. Set default name

Name is the first directive in the file, so let's start with that. The command line to set mine is git config --global user.name "Alan Formy-Duval". Instead of running this command, just edit the name directive in the configuration file:

name = Alan Formy-Duval3. Set default email address

The email address is the second directive, so let's update it. By default, Git uses your system-provided name and email address. If this is incorrect or you prefer something different, you can specify it in the configuration file. In fact, if you have not configured them, Git will let you know with a friendly message the first time you commit:

Committer: Alan <alan@hopper>
Your name and email address were configured automatically based
on your username and hostname. Please check that they are accurate....

The command line to set mine is git config --global user.email "alan@opensource.com". Instead, edit the email directive in the configuration file and provide your preferred address:

email = alan@opensource.com

The last two settings that I like to set are the default branch name and the default editor. These directives will need to be added while you are still in the editor.

4. Set default branch name

There is currently a trend to move away from the usage of the word master as the default branch name. As a matter of fact, Git will highlight this trend with a friendly message upon initialization of a new repository:

$ git init
hint: Using 'master' as the name for the initial branch. This default branch name
hint: is subject to change. To configure the initial branch name to use in all
hint: of your new repositories, which will suppress this warning, call:
hint:
hint:   git config --global init.defaultBranch <name>

This directive, named defaultBranch, needs to be located in a new section named init. It is now generally accepted that many coders use the word main for their default branch. This is what I like to use. Add this section followed by the directive to the configuration:

More Linux resources Linux commands cheat sheet Advanced Linux commands cheat sheet Free online course: RHEL technical overview Linux networking cheat sheet SELinux cheat sheet Linux common commands cheat sheet What are Linux containers? Our latest Linux articles [init]
            defaultBranch = main5. Set default editor

The fifth setting that I like to set is the default editor. This refers to the editor that Git will present for typing your commit message each time you commit changes to your repository. Everyone has a preference whether it is nano, emacs, vi, or something else. I'm happy with vi. So, to set your editor, add a core section that includes the editor directive:

[core]
            editor = vi

That's the last one. Exit the editor. Git saves the global configuration file in your home directory. If you run the editing command again, you will see all of the content. Notice that the configuration file is a plain text file, so it can also be viewed using text tools such as the cat command. This is how mine appears:

$ cat ~/.gitconfig
[user]
        email = alan@opensource.com
        name = Alan Formy-Duval
[core]
        editor = vi
[init]
        defaultBranch = main

This is a simple guide to quickly get started working with Git and a few of its many configuration options. There are many other articles on Git here at Opensource.com, as well as our downloadable Git cheat sheet.

This is a simple guide to quickly get started working with Git and a few of its many configuration options.

Linux Git What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

6 Python interpreters to try in 2022

Wednesday 21st of September 2022 07:00:00 AM
6 Python interpreters to try in 2022 Stephan Avenwedde Wed, 09/21/2022 - 03:00

Python, one of the most popular programming languages, requires an interpreter to execute the instructions defined by the Python code. In contrast to other languages, which compile directly into machine code, it’s up to the interpreter to read Python code and translate its instructions for the CPU performing the related actions. There are several interpreters out there, and in this article, I’ll take a look at a few of them.

Primer to interpreters

When talking about the Python interpreter, it’s usually the /usr/bin/python binary being referred to. That lets you execute a .py file.
However, interpreting is just one task. Before a line of Python code is actually executed on the CPU, these four steps are involved:

  1. Lexing - The human-made source code is converted into a sequence of logical entities, the so called lexical tokens.
  2. Parsing - In the parser, the lexical tokens are checked in regards of syntax and grammar. The output of the parser is an abstract syntax tree (AST).
  3. Compiling - Based on the AST, the compiler creates Python bytecode. The bytecode consists of very basic, platform independent instructions.
  4. Interpreting - The interpreter takes the bytecode and performs the specified operations.

As you can see, a lot of steps are required before any real action is taken. It makes sense to take a closer look at the different interpreters.

1. CPython

CPython is the reference implementation of Python and the default on many systems. As the name suggests, CPython is written in C.
As a result, it is possible to write extensions in C and therefore make the widley used C based library code available to Python. CPython is available on a wide range of platforms including ARM, iOS, and RISC. However, as the reference implementation of the language, CPython is carefully optimized and not focused on speed.

2. Pyston

Pyston is a fork of the CPython interpreter which implements performance optimizations. The project describes itself as a replacement of the standard CPython interpreter for large, real-world applications with a speedup potential up to 30%. Due to the lack of compatible binary packages, Pyston packages must be recompiled during the download process.

3. PyPy

PyPy is a Just-in-time (JIT) compiler for Python which is written in RPython, a statically typed subset of Python. In contrast to the CPython interpreter, PyPy compiles to machine code which can be directly executed by the CPU. PyPy is the playground for Python developers where they can experiment with new features more easily.

PyPy is faster than the reference CPython implementation. Because of the nature of JIT compiler, only applications that have been running for a long time benefit from caching.  PyPy can act as a replacement for CPython. There is a drawback, though. C-extension modules are mostly supported, but they run slower than a Python one. PyPy extension modules are written in Python (not C) and so the JIT compiler is able to optimized them. As long as your application isn't dependent on incompatible modules, PyPy is a great replacement for CPython. There is a dedicated page on the project website which describes the differences to CPython in detail: Diffrences between PyPy and CPython

4. RustPython

As the name suggest, RustPython is a Python interpreter written in Rust. Although the Rust programming language is quite new, it has been gaining popularity and is a candidate to be a successor of C and C++. By default, RustPython behaves like the interpreter of CPython but it also has a JIT compiler which can be enabled optionally. Another nice feature is that the Rust toolchain allows you to directly compile to WebAssembly and also allows you to run the interpreter completely in the browser. A demo of it can be found at rustpython.github.com/demo.

5. Stackless Python

Stackless Python describes itself as an enhanced version of the Python programming language. The project is basically a fork of the CPython interpreter which adds microthreads, channels and a scheduler to the language. Microthreads allow you to structure your code into tasklets which let you run your code in parallel. This approach is comparable to using green threads of the greenlet module. Channels can be used for bidirectional communication between tasklets. A famous user of Stackless Python is the MMORPG Eve Online.

More Python resources What is an IDE? Cheat sheet: Python 3.7 for beginners Top Python GUI frameworks Download: 7 essential PyPI libraries Red Hat Developers Latest Python articles 6. Micro Python

MicroPython is the way to go if you target micro controllers. It is a lean implementation that only requires 16kB of RAM and 256kB of space. Due to the embedded environment which it is intended for, MicroPython’s standard library is only a subset of CPython’s extensive STL. For developing and testing or as a lightweight alternative, MicroPython also runs on ordinary x86 and x64 machines. MicroPython is available for Linux, Windows, as well as many microcontrollers.

Performance

By design, Python is an inherently slow language. Depending on the task, there are significant performance differences between the interpreters. To get an overview of which interpreter is the best pick for a certain task, refer to pybenchmarks.org. An alternative to using an interpreter is to compile Python binary code directly into machine code. Nuitka, for example, is one of those projects which can compile Python code to C code and from C to machine code. The C code is then compiled to machine code using an ordinary C compiler. The topic of Python compilers is quite comprehensive and worth a separate article.

Summary

Python is a wonderful language for rapid prototyping and automating tasks. Additionally, it is easy to learn and well suited for beginners. If you usually stick with CPython, it could be interesting to see how your code behaves on another interpreter. If you use Fedora, you can easily test a few other interpreters as the package manager already provides the right binaries. Check out fedora.developer.org for more information.

It could be interesting to see how your code behaves on another interpreter than what you're used to.

Image by:

WOCinTech Chat. Modified by Opensource.com. CC BY-SA 4.0

Python Programming What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. 2 Comments Register or Login to post a comment. Greg Pittman | September 21, 2022 Register or Login to like

It's not quite clear, but I'm guessing that the default interpreter in Fedora is CPython?

Stephan Avenwedde | September 22, 2022 Register or Login to like

Hello Greg, yes, CPython is default.

In reply to by Greg Pittman

My favorite open source alternatives to Notion

Wednesday 21st of September 2022 07:00:00 AM
My favorite open source alternatives to Notion Amar Gandhi Wed, 09/21/2022 - 03:00

If you have notes to yourself scattered throughout your hard drive, you might need a notes application to collect and organize your personal reminders. A notes system can help you track ideas, important tasks, and works in progress. A popular application that isn't open source is Notion, but here are two options that respect your privacy and data.

Standard Notes

Standard Notes is an open source (AGPL 3.0 license) notes application featuring a password manager, a to-do list, and, of course, a great system for writing and storing notes.

One of the most important things about taking notes is finding them again, so organization is critical. Standard Notes uses an intuitive and natural tagging system to help you organize your content. You assign hashtags to each note to classify it.

Standard Notes is extensible through plug-ins. There are plug-ins for LaTeX, Markdown, code snippets, spreadsheets, and more. There's even an option to publish to a blogging platform, should you want to make some of your notes public.

Image by:

(Amir Gandhi, CC BY-SA 4.0)

Standard Notes also boasts numerous backup options, including email and cloud services. Furthermore, Standard Notes can work on any platform, including Linux, Windows, macOS, and Chrome OS.

More great content Free online course: RHEL technical overview Learn advanced Linux commands Download cheat sheets Find an open source alternative Explore open source resources Self-hosting Standard Notes

Standard Notes can be self-hosted. The developers provide a script that runs the application in a container, making it to run almost anywhere. If you've yet to explore containers, then you can get up to speed with Opensource.com's introduction to running applications in containers.

Another option is to use the hosted version provided by Standard Notes.

The development of Standard Notes can be followed on its Git repository.

Trilium

Trilium is a notes application that visually resembles Notion in many ways. It can handle various data types, including images, tables, to-do lists, highlighting, mind maps, flowcharts, family trees, code blocks, and more.

Trilium has several mechanisms to help you organize both your thoughts and your notes. You can view a history of recent changes, a global map of all your notes, note categories, or you can search for notes and contents.

Image by:

(Amir Gandhi, CC BY-SA 4.0)

You can install Trilium as a Flatpak from Flathub, or you can install it on your own server as a container. Alternatively, you can use Trilium's hosted instance.

Take note

There are plenty of useful note-taking applications in the open source world, and both Standard Notes and Trilium are designed with your data as the top priority. You can import and export data from these applications, so it's safe to try them out. You'll always have access to your data, so give Standard Notes or Trilium a try.

There are lots of useful open source note-taking tools out there. Standard Notes and Trilium are designed with your data as the top priority.

Alternatives What to read next 5 note-taking apps for Linux This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. 2 Comments Register or Login to post a comment. Yoka | September 22, 2022 Register or Login to like

I highly recommend checking out Logseq and Anytype as well. They are fantastic alternatives.

Brent Morris | September 24, 2022 Register or Login to like

I recently went through a process researching a good FOSS alternative to Notion as I use it daily but also hate how locked down it is.

I ended up choosing Logseq though it didn't quite have all the features I use (like databases). But I found a number of good candidates: Athens, TiddlyRoam, Zettlr, Simple note, Joplin.

I had a real focus on bidirectional linking and the concept of the Zettlekasten method as I feel that idea works well with how I think but even outside of that there's a number of great options.

Found a number of options that are not yet quite ready for prime time as it were, the most promising one I found was Affine which is a collaborative system that allows for linear block-based notes (like Notion) that can also be viewed as a kanban board (on arbitrary fields for grouping) and a whiteboard (like miro).

3 ways to use the Linux inxi command

Tuesday 20th of September 2022 07:00:00 AM
3 ways to use the Linux inxi command Don Watkins Tue, 09/20/2022 - 03:00

I was looking for information about the health of my laptop battery when I stumbled upon inxi. It's a command line system information tool that provides a wealth of information about your Linux computer, whether it's a laptop, desktop, or server.

The inxi command is licensed with the GPLv3, and many Linux distributions include it. According to its Git repository: "inxi strives to support the widest range of operating systems and hardware, from the most simple consumer desktops, to the most advanced professional hardware and servers."

Documentation is robust, and the project maintains a complete man page online. Once installed, you can access the man page on your system with the man inxi command.

More Linux resources Linux commands cheat sheet Advanced Linux commands cheat sheet Free online course: RHEL technical overview Linux networking cheat sheet SELinux cheat sheet Linux common commands cheat sheet What are Linux containers? Our latest Linux articles Install inxi on Linux

Generally, you can install inxi from your distribution's software repository or app center. For example, on Fedora, CentOS, Mageia, or similar:

$ sudo dnf install inxi

On Debian, Elementary, Linux Mint, or similar:

$ sudo apt install inxi

You can find more information about installation options for your Linux distribution here.

3 ways to use inxi on Linux

Once you install inxi, you can explore all its options. There are numerous options to help you learn more about your system. The most fundamental command provides a basic overview of your system:

$ inxi -b
System:
  Host: pop-os Kernel: 5.19.0-76051900-generic x86_64 bits: 64
        Desktop: GNOME 42.3.1 Distro: Pop!_OS 22.04 LTS
Machine:
  Type: Laptop System: HP product: Dev One Notebook PC v: N/A
        serial: <superuser required>
  Mobo: HP model: 8A78 v: KBC Version 01.03 serial: <superuser required>
        UEFI: Insyde v: F.05 date: 06/14/2022
Battery:
  ID-1: BATT charge: 50.6 Wh (96.9%) condition: 52.2/53.2 Wh (98.0%)
CPU:
  Info: 8-core AMD Ryzen 7 PRO 5850U with Radeon Graphics [MT MCP]
        speed (MHz): avg: 915 min/max: 400/4507
Graphics:
  Device-1: AMD Cezanne driver: amdgpu v: kernel
  Device-2: Quanta HP HD Camera type: USB driver: uvcvideo
  Display: x11 server: X.Org v: 1.21.1.3 driver: X: loaded: amdgpu,ati
        unloaded: fbdev,modesetting,radeon,vesa gpu: amdgpu
        resolution: 1920x1080~60Hz
  OpenGL:
        renderer: AMD RENOIR (LLVM 13.0.1 DRM 3.47 5.19.0-76051900-generic)
        v: 4.6 Mesa 22.0.5
Network:
  Device-1: Realtek RTL8822CE 802.11ac PCIe Wireless Network Adapter
        driver: rtw_8822ce
Drives:
  Local Storage: total: 953.87 GiB used: 75.44 GiB (7.9%)
Info:
  Processes: 347 Uptime: 15m Memory: 14.96 GiB used: 2.91 GiB (19.4%)
  Shell: Bash inxi: 3.3.131. Display battery status

You can check your battery health using the -B option. The result shows the system battery ID, charge condition, and other information:

$ inxi -B
Battery:
ID-1: BATT charge: 44.3 Wh (85.2%) condition: 52.0/53.2 Wh (97.7%)2. Display CPU info

Find out more information about the CPU with the -C option:

$ inxi -C CPU: Info: 8-core model: AMD Ryzen 7 PRO 5850U with Radeon Graphics bits: 64 type: MT MCP cache: L2: 4 MiB Speed (MHz): avg: 400 min/max: 400/4507 cores: 1: 400 2: 400 3: 400 4: 400 5: 400 6: 400 7: 400 8: 400 9: 400 10: 400 11: 400 12: 400 13: 400 14: 400 15: 400 16: 400

The output of inxi uses colored text by default. You can change that to improve readability, as needed, by using the "color switch."

The command option is -c followed by any number between 0 and 42 to suit your tastes.

$ inxi -c 42

Here is an example of a couple of different options using color 5 and then 7:

Image by:

(Don Watkins, CC BY-SA 4.0)

The software can show hardware temperature, fan speed, and other information about your system using the sensors in your Linux system. Enter inxi -s and read the result below:

Image by:

(Don Watkins, CC BY-SA 4.0)

3. Combine options

You can combine options for inxi to get complex output when supported. For example, inxi -S provides system information, and -v provides verbose output. Combining the two gives the following:

$ inxi -S
System:
  Host: pop-os Kernel: 5.19.0-76051900-generic x86_64 bits: 64
        Desktop: GNOME 42.3.1 Distro: Pop!_OS 22.04 LTS

$ inxi -Sv
CPU: 8-core AMD Ryzen 7 PRO 5850U with Radeon Graphics (-MT MCP-)
speed/min/max: 634/400/4507 MHz Kernel: 5.19.0-76051900-generic x86_64
Up: 20m Mem: 3084.2/15318.5 MiB (20.1%) Storage: 953.87 GiB (7.9% used)
Procs: 346 Shell: Bash inxi: 3.3.13Bonus: Check the weather

Your computer isn't all inxi can gather information about. With the -w option, you can also get weather information for your locale:

$ inxi -w
Weather:
  Report: temperature: 14 C (57 F) conditions: Clear sky
  Locale: Wellington, G2, NZL
        current time: Tue 30 Aug 2022 16:28:14 (Pacific/Auckland)
        Source: WeatherBit.io

You can get weather information for other areas of the world by specifying the city and country you want along with -W:

$ inxi -W rome,italy
Weather:
  Report: temperature: 20 C (68 F) conditions: Clear sky
  Locale: Rome, Italy current time: Tue 30 Aug 2022 06:29:52
        Source: WeatherBit.ioWrap up

There are many great tools to gather information about your computer. I use different ones depending on the machine, the desktop, or my mood. What are your favorite system information tools?

I use inxi on Linux to check my laptop battery, CPU information, and even the weather.

Linux Command line Hardware What to read next Linux commands to display your hardware information This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. 3 Comments Register or Login to post a comment. Digitalis | September 20, 2022 Register or Login to like

Nifty. Never heard of this before. Already installed on my system. Thank you!

Victorhck | September 20, 2022 Register or Login to like

Hello.
Spreading the word about this great free software tool in spanish:

https://victorhckinthefreeworld.com/2022/09/20/3-formas-de-utilizar-el-comando-inxi-en-gnu-linux/

inxi is a must to search info in internet about the system or hardware, or a way to get the info to share in forums, etc...

Greetings

Anderson Silva | September 20, 2022 Register or Login to like

Did you pick up an HP Dev One, Don? Nice article, I will try it out myself.

Security buzzwords to avoid and what to say instead

Tuesday 20th of September 2022 07:00:00 AM
Security buzzwords to avoid and what to say instead Seth Kenlon Tue, 09/20/2022 - 03:00

Technology is a little famous for coming up with "buzzwords." Other industries do it, too, of course. "Story-driven" and "rules light" tabletop games are a big thing right now, "deconstructed" burgers and burritos are a big deal in fine dining. The problem with buzzwords in tech, though, is that they potentially actually affect your life. When somebody calls an application "secure," to influence you to use their product, there's an implicit promise being made. "Secure" must mean that something's secure. It's safe for you to use and trust. The problem is, the word "secure" can actually refer to any number of things, and the tech industry often uses it as such a general term that it becomes meaningless.

Because "secure" can mean both so much and so little, it's important to use the word "secure" carefully. In fact, it's often best not to use the word at all, and instead, just say what you actually mean.

More on security The defensive coding guide 10 layers of Linux container security SELinux coloring book More security articles When "secure" means encrypted

Sometimes "secure" is imprecise shorthand for encrypted. In this context, "secure" refers to some degree of difficulty for outside observers to eavesdrop on your data.

Don't say this: "This website is resilient and secure."

That sounds pretty good! You're probably imagining a website that has several options for 2-factor authentication, zero-knowledge data storage, and steadfast anonymity policies.

Say this instead: "This website has a 99% uptime guarantee, and its traffic is encrypted and verifiable with SSL."

Not only is the intent of the promise clear now, it also explains how "secure" is achieved (it uses SSL) and what the scope of "secure" is.

Note that there's explicitly no promise here about privacy or anonymity.

When "secure" means restricted access

Sometimes the term "secure" refers to application or device access. Without clarification, "secure" could mean anything from the useless security by obscurity model, to a simple htaccess password, to biometric scanners.

Don't say this: "We've secured the system for your protection."

Say this instead: "Our system uses 2-factor authentication."

When "secure" means data storage

The term "secure" can also refer to the way your data is stored on a server or a device.

Don't say this: "This device stores your data with security in mind."

Say this instead: "This device uses full disk encryption to protect your data."

When remote storage is involved, "secure" may instead refer to who has access to stored data.

Don't say this: "Your data is secure."

Say this instead: "Your data is encrypted using PGP, and only you have the private key."

When "secure" means privacy

These days, the term "privacy" is almost as broad and imprecise as "security." On one hand, you might think that "secure" must mean "private," but that's true only when "secure" has been defined. Is something private because it has a password barrier to entry? Or is something private because it's been encrypted and only you have the keys? Or is it private because the vendor storing your data knows nothing about you (aside from an IP address?) It's not enough to declare "privacy" any more than it is to declare "security" without qualification.

Don't say this: "Your data is secure with us."

Say this instead: "Your data is encrypted with PGP, and only you have the private key. We require no personal data from you, and can only identify you by your IP address."

Some sites make claims about how long IP addresses are retained in logs, and promises about never surrendering data to authorities without warrants, and so on. Those are beyond the scope of technological "security," and have everything to do with trust, so don't confuse them for technical specifications.

Say what you mean

Technology is a complex topic with a lot of potential for confusion. Communication is important, and while shorthand and jargon can be useful in some settings, generally it's better to be precise. When you're proud of the "security" of your project, don't generalize it with a broad term. Make it clear to others what you're doing to protect your users, and make it equally clear what you consider out of scope, and communicate these things often. "Security" is a great feature, but it's a broad one, so don't be afraid to brag about the specifics.

Consider these thoughtful approaches to define what security really means in your open source project.

Image by:

JanBaby, via Pixabay CC0.

Security and privacy What to read next Why transparency is critical to your open source project's security Encrypting and decrypting files with OpenSSL This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

I got my first pull request merged!

Monday 19th of September 2022 07:00:00 AM
I got my first pull request merged! Oluwaseun Mon, 09/19/2022 - 03:00

Words cannot express my joy when I got the notification about the merge below, and I owe it to my current engineering school, AltSchool Africa.

Image by:

(Awosise Oluwaseun, CC BY-SA 4.0)

Before this, I had been introduced to open source many times, told about its importance in the tech space, and even attended open source conferences (e.g., OSCAFest). I had all the instant excitement to start, but imposter syndrome set in on opening GitHub to create something.

Fast forward to Monday, the 8th of August, 2022, when I watched Bolaji's video on contributing to open source. I felt pumped again, but I wanted to apply what I learned, so I noted some steps.

The steps:

  1. I made up my mind I was going to contribute to a project.
  2. I was focused on a site (good first issue) to pick my first project from, which I filtered to suit my skill level. I kept opening the next page till I found one.
  3. I made sure I was equipped with the required Git and GitHub knowledge to complete the project.

More great content Free online course: RHEL technical overview Learn advanced Linux commands Download cheat sheets Find an open source alternative Explore open source resources The project

After long hours searching for projects, I finally found one titled, Ensure no missing alt attributes. I was to give descriptive alt values to images from the site. Alt values in images help to improve the accessibility of the site such that screen readers can provide a detailed description of the image to, say, a visually impaired person. Easy right? Yes, but if I didn't make up my mind to get the first contribution, I wouldn't find it, and open source would continue to be a myth to me.

I was still pumped until I discovered it was from MDN. Wait, MDN? As in Mozilla developer? Will they merge my contribution even with how seemingly easy it looks? Imposter syndrome set in.

Upon checking the issue, I saw that people were already contributing. I summoned my courage and started reading about it. Taking my time to read and understand the project and how I needed to approach the issue was another challenge I had to overcome.

The project is as easy as you try to understand it.

So, I picked two images to begin with. I gave alt values to them, committed my changes, then made a pull request. The time between when I made the pull request and when I got the approval mail was full of self-doubts. Should I close the pull request? This is MDN. Well, it's not coding... What if I don't get merged? I might never contribute again. All it took to clear all of the doubts were the emails I got from my reviewer below:

Image by:

(Awosise Oluwaseun, CC BY-SA 4.0)

Image by:

(Awosise Oluwaseun, CC BY-SA 4.0)

Image by:

(Awosise Oluwaseun, CC BY-SA 4.0)

I was indeed delighted, and this inspired me to check for more. It gave me the courage I needed to request additional issues to solve.

Image by:

(Awosise Oluwaseun, CC BY-SA 4.0)

Summary

A few lessons I'd love you to take home from this article are:

  • Open source is for all. Do you see that typo on that site you just visited? You helping to correct it is a way of contributing.
  • No skillset is too small. A basic understanding of HTML was what I needed to contribute.
  • Only you can stop yourself from contributing.
  • The first contribution is all you need to get the ball rolling.

I hope you have been able to pick something from my story and apply it today. This is another space I'd like to keep contributing to, so see you in my next article, and happy open sourcing!

This article originally appeared on I got my first Pull Request merged! and is republished with permission.

Experience the joy that contributing to open source brings.

Image by:

Photo by Rob Tiller, CC BY-SA 4.0

Community management What to read next My first contribution to open source: Making a decision New open source tool catalogs African language resources This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

PyLint: The good, the bad, and the ugly

Monday 19th of September 2022 07:00:00 AM
PyLint: The good, the bad, and the ugly Moshe Zadka Mon, 09/19/2022 - 03:00

Hot take: PyLint is actually good!

"PyLint can save your life" is an exaggeration, but not as much as you might think! PyLint can keep you from really really hard to find and complicated bugs. At worst, it can save you the time of a test run. At best, it can help you avoid complicated production mistakes.

The good

I'm embarrassed to say how common this can be. Naming tests is perpetually weird: Nothing cares about the name, and there's often not a natural name to be found. For instance, look at this code:

def test_add_small():
    # Math, am I right?
    assert 1 + 1 == 3
   
def test_add_large():
    assert 5 + 6 == 11
   
def test_add_small():
    assert 1 + 10 == 11

The test works:

collected 2 items                                                                        
test.py ..
2 passed

More Python resources What is an IDE? Cheat sheet: Python 3.7 for beginners Top Python GUI frameworks Download: 7 essential PyPI libraries Red Hat Developers Latest Python articles But here's the kicker: If you override a name, the testing infrastructure happily skips over the test!

In reality, these files can be hundreds of lines long, and the person adding the new test might not be aware of all the names. Unless someone is looking at test output carefully, everything looks fine.

Worst of all, the addition of the overriding test, the breakage of the overridden test, and the problem that results in prod might be separated by days, months, or even years.

PyLint finds it

But like a good friend, PyLint is there for you.

test.py:8:0: E0102: function already defined line 1
     (function-redefined)The bad

Like a 90s sitcom, the more you get into PyLint, the more it becomes problematic. This is completely reasonable code for an inventory modeling program:

"""Inventory abstractions"""

import attrs

@attrs.define
class Laptop:
    """A laptop"""
    ident: str
    cpu: str

It seems that PyLint has opinions (probably formed in the 90s) and is not afraid to state them as facts:

$ pylint laptop.py | sed -n '/^laptop/s/[^ ]*: //p'
R0903: Too few public methods (0/2) (too-few-public-methods)The ugly

Ever wanted to add your own unvetted opinion to a tool used by millions? PyLint has 12 million monthly downloads.

"People will just disable the whole check if it's too picky." —PyLint issue 6987, July 3rd, 2022

The attitude it takes towards adding a test with potentially many false positives is..."eh."

Making it work for you

PyLint is fine, but you need to interact with it carefully. Here are the three things I recommend to make PyLint work for you.

1. Pin it

Pin the PyLint version you use to avoid any surprises!

In your .toml file:

[project.optional-dependencies]
pylint = ["pylint"]

In your code:

from unittest import mock

This corresponds with code like this:

# noxfile.py
...
@nox.session(python=VERSIONS[-1])
def refresh_deps(session):
    """Refresh the requirements-*.txt files"""
    session.install("pip-tools")
    for deps in [..., "pylint"]:
        session.run(
            "pip-compile",
            "--extra",
            deps,
            "pyproject.toml",
            "--output-file",
            f"requirements-{deps}.txt",
        )2. Default deny

Disable all checks. Then enable ones that you think have a high value-to-false-positive ratio. (Not just false-negative-to-false-positive ratio!)

# noxfile.py
...
@nox.session(python="3.10")
def lint(session):
    files = ["src/", "noxfile.py"]
    session.install("-r", "requirements-pylint.txt")
    session.install("-e", ".")
    session.run(
        "pylint",
        "--disable=all",
        *(f"--enable={checker}" for checker in checkers)
        "src",
    )3. Checkers

These are some of the ones I like. Enforce consistency in the project, avoid some obvious mistakes.

checkers = [
    "missing-class-docstring",
    "missing-function-docstring",
    "missing-module-docstring",
    "function-redefined",
]Using PyLint

You can take just the good parts of PyLint. Run it in CI to keep consistency, and use the highest value checkers.

Lose the bad parts: Default deny checkers.

Avoid the ugly parts: Pin the version to avoid surprises.

Get the most out of PyLint.

Image by:

Opensource.com

Python What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

What is OpenRAN?

Saturday 17th of September 2022 07:00:00 AM
What is OpenRAN? Stephan Avenwedde Sat, 09/17/2022 - 03:00

If you own and use a smartphone capable of connecting to arbitrary computers all over the world, then you are a user of Radio Access Networks (RAN). A RAN is provided by your cellular provider, and it handles wireless connections between your smartphone and your communication provider.

While your smartphone may be running an open source operating system (Android) and the server you try to access is probably running Linux, there's a lot of proprietary technology in between to make the connection happen. While you may have a basic understanding of how networking works locally, this knowledge stops when you plug a SIM card into your smartphone in order to make a connection with a cell tower possible. In fact, the majority of software and hardware components in and around a cell tower are still closed source, which of course has some drawbacks. This is where OpenRAN comes into play.

The OpenRAN initiative (shorthand for Open Radio Access Network) was started by the O-Ran Alliance, a worldwide community of mobile operators, vendors, and research and academic institutions. The initiative aims to define open standards between the various components of radio access networks. Interoperability between components of different manufacturers was not possible. Until now.

More on edge computing Understanding edge computing Why Linux is critical to edge computing eBook: Running Kubernetes on your Raspberry Pi Download now: The automated enterprise eBook eBook: A practical guide to home automation using open source tools eBook: 7 examples of automation on the edge The latest on edge Radio Access Network

But what exactly is a RAN? In a nutshell, a RAN establishes a wireless connection to devices (smartphones, for example) and connects them to the core network of the communication company. In the context of a RAN, devices are denoted as User Equipment (UE).

The tasks of a RAN can be summarized as follows:

  • Authentication of UE
  • The handover of UE to another RAN (if the UE is moving)
  • Forwarding the data between the UE and the core network
  • Provision of the data for accounting functions (billing of services or the transmitted data)
  • Control of access to the various services
OpenRAN

RAN usually consists of proprietary components. OpenRAN defines functional units and interfaces between them:

  • Radio Unit (RU): The RU is connected to the antenna and sends, receives, amplifies, and digitizes radio signals.
  • Distributed Unit (DU): Handles the PHY, MAC and RLC layer.
  • Centralised Unit (CU): Handles the RRC and PDCP layer.
  • RAN Intelligent Controller (RIC): Control and optimization of RAN elements and resources.

Units are connected to each other by standardized, open interfaces. Furthermore, if the units can be virtualized and deployed in the cloud or on an edge device, then it's called a vRAN (virtual Radio Access Network). The basic principle of vRAN is to decouple the hardware from the software by using a software-based virtualization layer. Using a vRAN improves flexibility in terms of scalability and the underlying hardware.

OpenRAN for everyone

By the definition of functional units and the interfaces between them, OpenRAN enables interoperability of components from different manufacturers. This reduces the dependency for cellular providers of specific vendors and makes communication infrastructure more flexible and resilient. As a side-effect, using clearly defined functional units and interfaces drives innovation and competition. With vRAN, the use of standard hardware is possible. With all these advantages, OpenRAN is a prime example of how open source benefits everyone.

Open Radio Access Network defines open standards between the various components of radio access networks.

Image by:

Opensource.com

Edge computing What to read next Streaming internet radio with RadioDroid PyRadio: An open source alternative for internet radio This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

Fix the apt-key deprecation error in Linux

Friday 16th of September 2022 07:00:00 AM
Fix the apt-key deprecation error in Linux Chris Hermansen Fri, 09/16/2022 - 03:00

This morning, after returning home from a mini vacation, I decided to run apt update and apt upgrade from the command line just to see whether there had been any updates while I was offline. After issuing the update command, something didn't seem quite right; I was seeing messages along the lines of:

W: https://updates.example.com/desktop/apt/dists/xenial/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.

True, it's just a warning, but still there's that scary word, deprecation, which usually means it's going away soon. So I thought I should take a look. Based on what I found, I thought my experience would be worth sharing.

It turns out that I have older configurations for some repositories, artifacts of installation processes from "back in the day," that needed adjustment. Taking my prompt from the warning message, I ran man apt-key at the command line, which provided several interesting bits of information. Near the beginning of the man page:

apt-key is used to manage the list of keys used by apt to authenticate packages. Packages which have been authenticated using these keys are considered trusted.
Use of apt-key is deprecated, except for the use of apt-key del in maintainer scripts to remove existing keys from the main keyring. If such usage of apt-key is desired, the additional installation of the GNU Privacy Guard suite (packaged in gnupg) is required.
apt-key(8) will last be available in Debian 11 and Ubuntu 22.04.

Last available in "Debian 11 and Ubuntu 22.04" is pretty much right now for me. Time to fix this!

More Linux resources Linux commands cheat sheet Advanced Linux commands cheat sheet Free online course: RHEL technical overview Linux networking cheat sheet SELinux cheat sheet Linux common commands cheat sheet What are Linux containers? Our latest Linux articles Fixing the apt-key deprecation error

Further on in the man page, there's the deprecation section mentioned in the warning from apt update:

DEPRECATION
Except for using apt-key del in maintainer scripts, the use of apt-key is deprecated. This section shows how to replace the existing use of apt-key.
If your existing use of apt-key add looks like this:

wget -qO- https://myrepo.example/myrepo.asc | sudo apt-key add -

Then you can directly replace this with (though note the recommendation below):

wget -qO- https://myrepo.example/myrepo.asc | sudo tee /etc/apt/trusted.gpg.d/myrepo.asc

Make sure to use the "asc" extension for ASCII armored keys and the "gpg" extension for the binary OpenPGP format (also known as "GPG key public ring"). The binary OpenPGP format works for all apt versions, while the ASCII armored format works for apt version >= 1.4.

Recommended: Instead of placing keys into the /etc/apt/trusted.gpg.d directory, you can place them anywhere on your filesystem by using the Signed-By option in your sources.list and pointing to the filename of the key. See sources.list(5) for details. Since APT 2.4, /etc/apt/keyrings is provided as the recommended location for keys not managed by packages. When using a deb822-style sources.list, and with apt version >= 2.4, the Signed-By option can also be used to include the full ASCII armored keyring directly in the sources.list without an additional file.

If you, like me, have keys from non-repository stuff added with apt-key, then here are the steps to transition:

  1. Determine which keys are in apt-key keyring /etc/apt/trusted.gpg
  2. Remove them
  3. Find and install replacements in /etc/apt/trusted.gpg.d/ or in /etc/apt/keyrings/
1. Finding old keys

The command apt-key list shows the keys in /etc/apt/trusted.gpg:

$ sudo apt-key list
[sudo] password:
Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)).
/etc/apt/trusted.gpg
--------------------
pub   rsa4096 2017-04-05 [SC]
      DBE4 6B52 81D0 C816 F630  E889 D980 A174 57F6 FB86
uid           [ unknown] Example <support@example.com>
sub   rsa4096 2017-04-05 [E]

pub   rsa4096 2016-04-12 [SC]
      EB4C 1BFD 4F04 2F6D DDCC  EC91 7721 F63B D38B 4796
uid           [ unknown] Google Inc. (Linux Packages Signing Authority) <linux-packages-keymaster@google.com>
sub   rsa4096 2021-10-26 [S] [expires: 2024-10-25]
[...]

Also shown afterward are the keys held in files in the /etc/apt/trusted.gpg.d folder.

[ Related read How to import your existing SSH keys into your GPG key ]

2. Removing old keys

The group of quartets of hex digits, for example DBEA 6B52...FB86, is the identifier required to delete the unwanted keys:

$ sudo apt-key del "DBEA 6B52 81D0 C816 F630  E889 D980 A174 57F6 FB86"

This gets rid of the Example key. That's literally just an example, and in reality you'd get rid of keys that actually exist. For instance, I ran the same command for each of the real keys on my system, including keys for Google, Signal, and Ascensio. Keys on your system will vary, depending on what you have installed.

3. Adding keys

Getting the replacement keys is dependent on the application. For example, Open Whisper offers its key and an explanation of what to do to install it, which I decided not to follow as it puts the key in /usr/share/keyrings. Instead, I did this:

$ wget -O- https://updates.signal.org/desktop/apt/keys.asc | gpg --dearmor > signal-desktop-keyring.gpg
$ sudo mv signal-desktop-keyring.gpg /etc/apt/trusted.gpg.d/
$ sudo chown root:root /etc/apt/trusted.gpg.d/signal-desktop-keyring.gpg
$ sudo chmod ugo+r /etc/apt/trusted.gpg.d/signal-desktop-keyring.gpg
$ sudo chmod go-w /etc/apt/trusted.gpg.d/signal-desktop-keyring.gpg

Ascencio also offers instructions for installing OnlyOffice that include dealing with the GPG key. Again I modified their instructions to suit my needs:

$ gpg --no-default-keyring --keyring gnupg-ring:~/onlyoffice.gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys CB2DE8E5
$ sudo mv onlyoffice.gpg /etc/apt/trusted.gpg.d/
$ sudo chown root:root /etc/apt/trusted.gpg.d/onlyoffice.gpg
$ sudo chmod ugo+r /etc/apt/trusted.gpg.d/onlyoffice.gpg
$ sudo chmod go-w /etc/apt/trusted.gpg.d/onlyoffice.gpg

As for the Google key, it is managed (correctly, it appears) through the .deb package, and so a simple reinstall with dpkg -i was all that was needed. Finally, I ended up with this:

$ ls -l /etc/apt/trusted.gpg.d
total 24
-rw-r--r-- 1 root root 7821 Sep  2 10:55 google-chrome.gpg
-rw-r--r-- 1 root root 2279 Sep  2 08:27 onlyoffice.gpg
-rw-r--r-- 1 root root 2223 Sep  2 08:02 signal-desktop-keyring.gpg
-rw-r--r-- 1 root root 2794 Mar 26  2021 ubuntu-keyring-2012-cdimage.gpg
-rw-r--r-- 1 root root 1733 Mar 26  2021 ubuntu-keyring-2018-archive.gpgExpired keys

The last problem key I had was from an outdated installation of QGIS. The key had expired, and I'd set it up to be managed by apt-key. I ended up following their instructions to the letter, both for installing a new key in /etc/apt/keryings and their suggested format for the /etc/apt/sources.list.d/qgis.sources installation configuration.

[ Download the Linux cheat sheets for apt or dnf ]

Linux system maintenance

Now you can run apt update with no warnings or errors related to deprecated key configurations. We apt users just need to remember to adjust any old installation instructions that depend on apt-key. Instead of using apt-key, you must instead install a key to /etc/apt/trusted.gpg.d/ or /etc/apt/keyrings/, using gpg as needed.

Follow these steps and you can run apt update with no warnings or errors related to deprecated key configurations.

Image by:

Opensource.com

Linux Sysadmin What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

24 of our favorite articles in a downloadable eBook

Thursday 15th of September 2022 07:00:00 AM
24 of our favorite articles in a downloadable eBook Lauren Pritchett Thu, 09/15/2022 - 03:00

One day in March of 2020, a few of my Opensource.com teammates and I grabbed lunch to talk about how we would work from home for the next couple of weeks until this pandemic got nipped in the bud. At the end of the work day, I packed up my laptop and walked out the door of our office building. Several months later, we were all still working from home. None of us had returned to our office-based workstations. In July 2020, using safety precautions, we were granted access to our workstation solely to retrieve personal items. My desk was left exactly as I left it that afternoon in March. Expiring snacks stashed in my secret drawer. Picture frames collecting dust. Comic strips pinned up.

And a pile of several bound copies of Best of a decade on Opensource.com 2010-2019. Our last yearbook that was published (and printed)!

Like most folks, the Opensource.com community had to pivot how we operated in order to stay connected. Sure, we continued our weekly video calls. But in-person conferences, a unique time where people would travel from all over the world to be together, were out of the question. Though some of this operational stuff has changed, the connection with one another has strengthened. It is due time to publish a new yearbook to honor that connection. This yearbook was created to celebrate our correspondents.

The Opensource.com Correspondent Program recognizes the critical group of our most trusted and committed contributors. We recently closed out yet another successful program year with 24 correspondents. Each correspondent selected their favorite article to be included in this downloadable yearbook. In it, you'll find Raspberry Pi tutorials, career stories, home automation tips, Linux tricks, and much more.

Image by:

Opensource.com, CC-BY-4.0

The above screenshot is from a recent video call with a few of our correspondents. This community reaches far in locale (North Carolina, Minnesota, California, Germany, India, and New Zealand to name a few) and wide in experience (educators, hobbyists, sysadmins, podcasters, and more).

I'm looking forward to spending another year with the Opensource.com community, whether that is through video calls, in-person events, an Internet DM, or simply reading your article. How would you like to participate?

Click here to download the Opensource.com Correspondent Yearbook 2022 ]

The Opensource.com Correspondent Yearbook 2021-2022 features articles about Raspberry Pi, Linux, open source careers, programming, and more.

Image by:

Monsterkoi. Modified by Opensource.com. CC BY-SA 4.0

Opensource.com community What to read next New year, new Opensource.com community manager This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. 4 Comments Register or Login to post a comment. Ritanshu Tiwari | September 15, 2022 Register or Login to like

Great job.

Don Watkins | September 15, 2022 Register or Login to like

Thanks Lauren! Lots of changes in the past nearly three years. You've captured them well. The weekly correspondent meetings were a light in a dark time. The best part of being a correspondent is the community we belong to.

Hasan | September 18, 2022 Register or Login to like

I'd love to be here too, sorry I don't know enough English to speak. :(

AmyJune Hineline | September 19, 2022 Register or Login to like

Hasen, you don't have to be a native English speaker to be in the program.

https://opensource.com/writers

In reply to by Hasan

How I switched from Docker Desktop to Colima

Thursday 15th of September 2022 07:00:00 AM
How I switched from Docker Desktop to Colima Michael Anello Thu, 09/15/2022 - 03:00

DDEV is an open source tool that makes it simple to get local PHP development environments up and running within minutes. It’s powerful and flexible as a result of its per-project environment configurations, which can be extended, version controlled, and shared. In short, DDEV aims to allow development teams to use containers in their workflow without the complexities of bespoke configuration.

DDEV replaces more traditional AMP stack solutions (WAMP, MAMP, XAMPP, and so on) with a flexible, modern, container-based solution. Because it uses containers, DDEV allows each project to use any set of applications, versions of web servers, database servers, search index servers, and other types of software.

In March 2022, the DDEV team announced support for Colima, an open source Docker Desktop replacement for macOS and Linux. Colima is open source, and by all reports it’s got performance gains over its alternative, so using Colima seems like a no-brainer.

Migrating to Colima

First off, Colima is almost a drop-in replacement for Docker Desktop. I say almost because some reconfiguration is required when using it for an existing DDEV project. Specifically, databases must be reimported. The fix is to first export your database, then start Colima, then import it. Easy.

Colima requires that either the Docker or Podman command is installed. On Linux, it also requires Lima.

Docker is installed by default with Docker Desktop for macOS, but it’s also available as a stand-alone command. If you want to go 100% pure Colima, you can uninstall Docker Desktop for macOS, and install and configure the Docker client independently. Full installation instructions can be found on the DDEV docs site.

Image by:

(Mike Anello,CC BY-SA 4.0)

If you choose to keep using both Colima and Docker Desktop, then when issuing docker commands from the command line, you must first specify which container you want to work with. More on this in the next section.

More on Kubernetes What is Kubernetes? Free online course: Containers, Kubernetes and Red Hat OpenShift technical over… eBook: Storage Patterns for Kubernetes Test drive OpenShift hands-on An introduction to enterprise Kubernetes How to explain Kubernetes in plain terms eBook: Running Kubernetes on your Raspberry Pi homelab Kubernetes cheat sheet eBook: A guide to Kubernetes for SREs and sysadmins Latest Kubernetes articles Install Colima on macOS

I currently have some local projects using Docker, and some using Colima. Once I understood the basics, it’s not too difficult to switch between them.

  1. To get started, install Colima using Homebrew brew install colima

  2. ddev poweroff (just to be safe)

  3. Next, start Colima with colima start --cpu 4 --memory 4. The --cpu and --memory options only have to be done once. After the first time, only colima start is necessary.

  4. If you’re a DDEV user like me, then you can spin up a fresh Drupal 9 site with the usual ddev commands (ddev config, ddev start, and so on.) It’s recommended to enable DDEV’s mutagen functionality to maximize performance.

Switching between a Colima and Docker Desktop

If you’re not ready to switch to Colima wholesale yet, it’s possible to have both Colima and Docker Desktop installed.

  1. First, poweroff ddev:ddev poweroff

  2. Then stop Colima: colima stop

  3. Now run docker context use default to tell the Docker client which container you want to work with. The name default refers to Docker Desktop for Mac. When colima start is run, it automatically switches Docker to the colima context.

  4. To continue with the default (Docker Desktop) context, use the ddev start command.

Technically, starting and stopping Colima isn’t necessary, but the ddev poweroff command when switching between two contexts is.

Recent versions of Colima revert the Docker context back to default when Colima is stopped, so the docker context use default command is no longer necessary. Regardless, I still use docker context show to verify that either the default (Docker Desktop for Mac) or colima context is in use. Basically, the term context refers to which container provider the Docker client routes commands to.

Try Colima

Overall, I’m liking what I see so far. I haven’t run into any issues, and Colima-based sites seem a bit snappier (especially when DDEV’s Mutagen functionality is enabled). I definitely foresee myself migrating project sites to Colima over the next few weeks.

This article originally appeared on the DrupalEasy blog and is republished with permission.

Colima is a Docker Desktop alternative for macOS and Linux that's now supported by DDEV.

Image by:

freephotocc via Pixabay CC0

Kubernetes Mac Linux Drupal What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

3 steps to protect your home network

Wednesday 14th of September 2022 07:00:00 AM
3 steps to protect your home network Seth Kenlon Wed, 09/14/2022 - 03:00

The typical setup for Internet connectivity today is for your home to have a router, usually a little physical box located somewhere in your house, that acts as a gateway to the rest of the world. The router creates a local network, and you connect your devices to it, including your computer, mobile, TV, game console, and anything else that needs to connect to the Internet or to each other. It's deceptively easy to think of this setup as there being two "sides" of your router: On one side there's the Internet, and on the other, your devices. That's an awful colloquial, though, because in reality there's an entire worldwide network of computers on one side of your router, and your digital life on the other. When you use the Internet directly, you're logging onto a shared area of somebody else's computer. When you're not using the Internet, it doesn't go away, and there are lots of scripts and programs out there designed to visit millions upon millions of routers in an attempt to find open ports or services. With the Internet of Things (IoT) commonplace, there are sometimes more services running on your home network than you realize. Here are three steps you can take to audit and protect your home network from unwanted traffic.

[ Related read Run your network with open source software ]

1. Think about protocol

Part of your router's job is to keep the Internet separate from your home network. But when you access the Internet, you invite some portion of the Internet into your home. You're making an exception to the general rule that the Internet should stay off your network.

On many websites, what's allowed through your router is just text. When you visit your favorite blog site to read up on the latest tech news, for instance, you're downloading a page or two of text. You read the text, and then move on. That's a simple one-to-one transaction.

However, the HTTPS protocol is robust and the applications running on the Internet are full of variety. When you visit Opensource.com, for instance, you're not just downloading text. You get graphics, and maybe a cheat sheet or ebook. You're also downloading cookies in the background, which helps site administrators understand who visits the site, which has led to improved mobile support, a new design for greater accessibility, and content that readers enjoy. You may not think about cookies or traffic analysis as something you interact with when you're on the Internet, but it's something that gets "snuck" into page interactions because the HTTPS protocol is designed to be broad and, in many ways, high trust. When you visit a website over HTTPS (that is, in a web browser), you're implicitly agreeing to automatic downloads of files that you're probably not conscious of, but that you trust are useful and unobtrusive. For a model of file sharing designed for less trust, you might try the Gemini or Gopher space.

You make a similar agreement when you join a video conference. Not only are you downloading text on the page, cookies for traffic monitoring, but also a video and audio feed.

Some sites are designed for even more. There are sites designed to allow people to share their computer screen, and sometimes even the control of their computer. In the best case scenario, this helps a remote technician repair a problem on someone's computer, but in practice users can be tricked into visiting sites only to have financial credentials and personal data stolen.

You'd rightfully be suspicious if a website offering text articles required you to grant it permission to look through your webcam while you read. You should cultivate the same level of suspicion when an appliance requires Internet access. When you connect a device to your network, it's important to consider what implicit agreement you're making. A device designed to control lighting in your house shouldn't require Internet access to function, but many do, and many don't make it clear what permissions you're granting that device. Many IoT devices want access to the Internet so that you can access the device over the Internet while you're away from home. That's part of the appeal of a "smart home". However, it's impossible to know what code many of these devices run. When possible, use open source and trusted software, such as Home Assistant to interface with your living space.

[ Also read How to choose a wireless protocol for home automation ]

More on edge computing Understanding edge computing Why Linux is critical to edge computing eBook: Running Kubernetes on your Raspberry Pi Download now: The automated enterprise eBook eBook: A practical guide to home automation using open source tools eBook: 7 examples of automation on the edge The latest on edge 2. Create a guest network

Many modern routers make it trivial to create a second network (usually called a "guest network" in the configuration panels) for your home. You probably don't feel like you need a second network, but actually a guest network can be useful to have around. Its eponymous and most obvious use case is that a guest network provides people visiting your house access to the Internet without you telling them your network password. In the foyer of my house, I have a sign identifying the guest network name and password. Anyone who visits can join that network for access to the Internet.

The other use case is for IoT, edge devices, and my home lab. When I purchased "programmable" Christmas lights last year, I was surprised to find that in order to connect to the lights, they had to be connected to the Internet. OF course, the $50 lights from a nameless factory didn't come with source code included, or any way to interface or inspect with the firmware embedded in the power brick, and so I wasn't confident in what I was agreeing to by connecting them to the Internet. They've been permanently relegated to my guest network.

Every router vendor is different, so there's no single instruction on how to create a "sandboxed" guest network on yours. Generally, you access your home router through a web browser. Your router's address is sometimes printed on the bottom of the router, and it begins with either 192.168 or 10.

Navigate to your router's address and log in with the credentials you were provided when you got your Internet service. It's often as simple as admin with a numeric password (sometimes, this password is printed on the router, too). If you don't know the login, call your Internet provider and ask for details.

In the graphical interface, find the panel for "Guest network." This option is in the Advanced configuration of my router, but it could be somewhere else on yours, and it may not even be called "Guest network" (or it may not even be an option.)

Image by:

(Opensource.com, CC BY-SA 4.0)

It may take a lot of clicking around and reading. If you find that you have the option, then you can set up a guest network for visitors, including people walking through your front door and applications running on a lightbulb.

3. Firewall your firewall

Your router probably has a firewall running by default. A firewall keeps unwanted traffic off your network, usually by limiting incoming packets to HTTP and HTTPS (web browser traffic) and a few other utility protocols, and by rejecting traffic you didn't initiate. You can verify that a firewall is running by logging onto your router and looking for "Firewall" or "Security" settings.

However, many devices can run firewalls of there own. This is important because a network is a network because devices connect to one another. Placing firewalls "between" devices is like locking a door to a room inside your house. Guests may roam the halls, but without the right key they're not invited into your office.

On Linux, you can configure your firewall using firewalld interface and the firewall-cmd command. On other operating systems, the firewall is sometimes in a control panel labeled as "security" or "sharing" (and sometimes both.) Most default firewall settings allow only outgoing traffic (that's the traffic you initiate by, for instance, opening a browser and navigating to a website) and incoming traffic that's responding to your requests (that's the web data responding to your navigation). Incoming traffic that you didn't initiate is blocked.

You can customize this setup as needed, should you want to allow specific traffic, such as an SSH connection, a VNC connection, or a game server host.

Monitor your network

These techniques help build up your awareness of what's happening around you. The next step is to monitor your network. You can start simple, for instance by running Fail2ban on a test server on your guest network. Take a look at logs, if your router provides them. You don't have to know everything about TCP/IP and packets and other advanced subjects to see that the Internet is a busy and noisy place, and seeing that for yourself is great inspiration to take precautions when you set up a new device, whether it's IoT, mobile, a desktop or laptop, a game console, or a Raspberry Pi, in your home.

Who has access to your home network? With the Internet of Things (IoT) commonplace, there are sometimes more services running on your home network than you realize. Protect it from unwanted traffic.

Image by:

Opensource.com

Edge computing Networking Internet of Things (IoT) Security and privacy What to read next A beginner's guide to network management Control your home automation remotely with Raspberry Pi and Traefik Hub This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

Packaging Job scripts in Kubernetes operators

Wednesday 14th of September 2022 07:00:00 AM
Packaging Job scripts in Kubernetes operators Bobby Gryzynger Wed, 09/14/2022 - 03:00

When using a complex Kubernetes operator, you often have to orchestrate Jobs to perform workload tasks. Examples of Job implementations typically provide trivial scripts written directly in the manifest. In any reasonably-complex application, however, determining how to handle more-than-trivial scripts can be challenging.

In the past, I've tackled this problem by including my scripts in an application image. This approach works well enough, but it does have a drawback. Anytime changes are required, I'm forced to rebuild the application image to include the revisions. This is a lot of time wasted, especially when my application image takes a significant amount of time to build. This also means that I'm maintaining both an application image and an operator image. If my operator repository doesn't include the application image, then I'm making related changes across repositories. Ultimately, I'm multiplying the number of commits I make, and complicating my workflow. Every change means I have to manage and synchronize commits and image references between repositories.

More on Kubernetes What is Kubernetes? Free online course: Containers, Kubernetes and Red Hat OpenShift technical over… eBook: Storage Patterns for Kubernetes Test drive OpenShift hands-on An introduction to enterprise Kubernetes How to explain Kubernetes in plain terms eBook: Running Kubernetes on your Raspberry Pi homelab Kubernetes cheat sheet eBook: A guide to Kubernetes for SREs and sysadmins Latest Kubernetes articles

Given these challenges, I wanted to find a way to keep my Job scripts within my operator's code base. This way, I could revise my scripts in tandem with my operator's reconciliation logic. My goal was to devise a workflow that would only require me to rebuild the operator's image when I needed to make revisions to my scripts. Fortunately, I use the Go programming language, which provides the immensely helpful go:embed feature. This allows developers to package text files in with their application's binary. By leveraging this feature, I've found that I can maintain my Job scripts within my operator's image.

Embed Job script

For demonstration purposes, my task script doesn't include any actual business logic. However, by using an embedded script rather than writing the script directly into the Job manifest, this approach keeps complex scripts both well-organized and abstracted from the Job definition itself.

Here's my simple example script:

$ cat embeds/task.sh
#!/bin/sh
echo "Starting task script."
# Something complicated...
echo "Task complete."

Now to work on the operator's logic.

Operator logic

Here's the process within my operator's reconciliation:

  1. Retrieve the script's contents
  2. Add the script's contents to a ConfigMap
  3. Run the ConfigMap's script within the Job by
    1. Defining a volume that refers to the ConfigMap
    2. Making the volume's contents executable
    3. Mounting the volume to the Job 

Here's the code:

// STEP 1: retrieve the script content from the codebase.
//go:embed embeds/task.sh
var taskScript string

func (r *MyReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
        ctxlog := ctrllog.FromContext(ctx)
        myresource := &myresourcev1alpha.MyResource{}
        r.Get(ctx, req.NamespacedName, d)

        // STEP 2: create the ConfigMap with the script's content.
        configmap := &corev1.ConfigMap{}
        err := r.Get(ctx, types.NamespacedName{Name: "my-configmap", Namespace: myresource.Namespace}, configmap)
        if err != nil && apierrors.IsNotFound(err) {

                ctxlog.Info("Creating new ConfigMap")
                configmap := &corev1.ConfigMap{
                        ObjectMeta: metav1.ObjectMeta{
                                Name:      "my-configmap",
                                Namespace: myresource.Namespace,
                        },
                        Data: map[string]string{
                                "task.sh": taskScript,
                        },
                }

                err = ctrl.SetControllerReference(myresource, configmap, r.Scheme)
                if err != nil {
                        return ctrl.Result{}, err
                }
                err = r.Create(ctx, configmap)
                if err != nil {
                        ctxlog.Error(err, "Failed to create ConfigMap")
                        return ctrl.Result{}, err
                }
                return ctrl.Result{Requeue: true}, nil
        }

        // STEP 3: create the Job with the ConfigMap attached as a volume.
        job := &batchv1.Job{}
        err = r.Get(ctx, types.NamespacedName{Name: "my-job", Namespace: myresource.Namespace}, job)
        if err != nil && apierrors.IsNotFound(err) {

                ctxlog.Info("Creating new Job")
                configmapMode := int32(0554)
                job := &batchv1.Job{
                        ObjectMeta: metav1.ObjectMeta{
                                Name:      "my-job",
                                Namespace: myresource.Namespace,
                        },
                        Spec: batchv1.JobSpec{
                                Template: corev1.PodTemplateSpec{
                                        Spec: corev1.PodSpec{
                                                RestartPolicy: corev1.RestartPolicyNever,
                                                // STEP 3a: define the ConfigMap as a volume.
                                                Volumes: []corev1.Volume{{
                                                        Name: "task-script-volume",
                                                        VolumeSource: corev1.VolumeSource{
                                                                ConfigMap: &corev1.ConfigMapVolumeSource{
                                                                        LocalObjectReference: corev1.LocalObjectReference{
                                                                                Name: "my-configmap",
                                                                        },
                                                                        DefaultMode: &configmapMode,
                                                                },
                                                        },
                                                }},
                                                Containers: []corev1.Container{
                                                        {
                                                                Name:  "task",
                                                                Image: "busybox",
                                                                Resources: corev1.ResourceRequirements{
                                                                        Requests: corev1.ResourceList{
                                                                                corev1.ResourceCPU:    *resource.NewMilliQuantity(int64(50), resource.DecimalSI),
                                                                                corev1.ResourceMemory: *resource.NewScaledQuantity(int64(250), resource.Mega),
                                                                        },
                                                                        Limits: corev1.ResourceList{
                                                                                corev1.ResourceCPU:    *resource.NewMilliQuantity(int64(100), resource.DecimalSI),
                                                                                corev1.ResourceMemory: *resource.NewScaledQuantity(int64(500), resource.Mega),
                                                                        },
                                                                },
                                                                // STEP 3b: mount the ConfigMap volume.
                                                                VolumeMounts: []corev1.VolumeMount{{
                                                                        Name:      "task-script-volume",
                                                                        MountPath: "/scripts",
                                                                        ReadOnly:  true,
                                                                }},
                                                                // STEP 3c: run the volume-mounted script.
                                                                Command: []string{"/scripts/task.sh"},
                                                        },
                                                },
                                        },
                                },
                        },
                }

                err = ctrl.SetControllerReference(myresource, job, r.Scheme)
                if err != nil {
                        return ctrl.Result{}, err
                }
                err = r.Create(ctx, job)
                if err != nil {
                        ctxlog.Error(err, "Failed to create Job")
                        return ctrl.Result{}, err
                }
                return ctrl.Result{Requeue: true}, nil
        }

        // Requeue if the job is not complete.
        if *job.Spec.Completions == 0 {
                ctxlog.Info("Requeuing to wait for Job to complete")
                return ctrl.Result{RequeueAfter: time.Second * 15}, nil
        }

        ctxlog.Info("All done")
        return ctrl.Result{}, nil
}

After my operator defines the Job, all that's left to do is wait for the Job to complete. Looking at my operator's logs, I can see each step in the process recorded until the reconciliation is complete:

2022-08-07T18:25:11.739Z  INFO  controller.myresource   Creating new ConfigMap  {"reconciler group": "myoperator.myorg.com", "reconciler kind": "MyResource", "name": "myresource-example", "namespace": "default"}
2022-08-07T18:25:11.765Z  INFO  controller.myresource   Creating new Job        {"reconciler group": "myoperator.myorg.com", "reconciler kind": "MyResource", "name": "myresource-example", "namespace": "default"}
2022-08-07T18:25:11.780Z  INFO  controller.myresource   All done        {"reconciler group": "myoperator.myorg.com", "reconciler kind": "MyResource", "name": "myresource-example", "namespace": "default"}Go for Kubernetes

When it comes to managing scripts within operator-managed workloads and applications, go:embed provides a useful mechanism for simplifying the development workflow and abstracting business logic. As your operator and its scripts become more complex, this kind of abstraction and separation of concerns becomes increasingly important for the maintainability and clarity of your operator.

Embed scripts into your Kubernetes operators with Go.

Kubernetes Go programming language What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

How I troubleshoot swappiness and startup time on Linux

Tuesday 13th of September 2022 07:00:00 AM
How I troubleshoot swappiness and startup time on Linux David Both Tue, 09/13/2022 - 03:00

I recently experienced another interesting problem in the Linux startup sequence that has a circumvention–not a solution. It started quite unexpectedly.

I was writing a couple of articles while making some updates to my personal copy of my series of books, "Using and Administering Linux: Zero to SysAdmin." I had four instances of LibreOffice Write open to doing all that. I had three VMs running with VirtualBox to test some of the things I was writing about. I also had LibreOffice Impress open to work on an unrelated presentation. I like to listen to music, so I had one of several tabs in Firefox open to Pandora, my music streaming service of choice. I had multiple Bash shells open using Konsole with numerous tabs and the Alpine text-mode email client in one. Then there were the various tabs in the Thunar file manager.

So I had a lot going on. Just like I do now as I write this article.

The symptoms

As I used these open sessions, I noticed that things slowed down considerably while waiting for the system to write a document to the M.3 SSD–a process that should have been really fast. I also noticed that the music was choppy and dropped out completely every few minutes. Overall performance was generally poor. I began to think that Fedora had a serious problem.

My primary workstation, the one I was working on at the time, has 64GB of RAM and an Intel Core i9 Extreme with 16 cores and Hyperthreading (32 CPUs) that can run as fast as 4.1 GHz using my configured overclocking. So I should not have experienced any slowdowns–or so I thought at the time.

Determine the problem

It did not take long to find the problem because I have experienced similar symptoms before on systems with far less memory. The issue looked like delays due to page swapping. But why?

I started with one of my go-to tools for problem determination, htop. It showed that the system was using 13.6GB of memory for programs, and most of the rest of the RAM was in cache and buffers. It also showed that swapping was actively occurring and that about 253MB of data was stored in the swap partitions.

Date & Time: 2022-08-12 10:53:08
Uptime: 2 days, 23:47:15
Tasks: 200, 1559 thr, 371 kthr; 4 running
Load average: 3.97 3.05 2.08
   
Disk IO: 202.6% read: 687M write: 188K
Network: rx: 0KiB/s tx: 0KiB/s (0/0 packets)
Systemd: running (0/662 failed) (0/7912 jobs)    
Mem[|||||||##*@@@@@@@@@@@@@@@@@@@@@@@@@@    13.6G/62.5G]
Swp[||#                                      253M/18.0G]

But that meant I still had lots of memory left the system could use directly for programs and data and more that it could recover from cache and buffers. So why was this system even swapping at all?

I remembered hearing about the "swappiness" factor in one of my Red Hat training classes. But that was a long time ago. I did some searches on "swappiness" to learn about the kernel setting vm.swappiness.

The default setting for this kernel parameter is 60. That number is an abstract value that represents how aggressively the kernel tries to swap. Contrary to common but erroneous understanding – including mine before revising this article – this number does not represent a percentage of RAM. The value of vm.swappiness is used in a formula that determines multiple aspects of how swap is performed by the Linux kernel.

Based on my online reading, I discovered that 10% is a better value for vm.swappiness for many Linux systems with large amounts of RAM. I checked the current swappiness setting on my system and it was set to the default.

# sysctl vm.swappiness
vm.swappiness = 60

Time to change this kernel setting.

More for sysadmins Enable Sysadmin blog The Automated Enterprise: A guide to managing IT with automation eBook: Ansible automation for Sysadmins Tales from the field: A system administrator's guide to IT automation eBook: A guide to Kubernetes for SREs and sysadmins Latest sysadmin articles Fix the issue

I won't dive into the gory details, but the bottom line is that either of the following commands, run as root, will instantly do the job on a running Linux computer without a reboot.

# sysctl -w vm.swappiness=10

You could also use this next command to do the same thing.

# echo 10 > /proc/vm/swappiness

Tecmint has an excellent article about setting kernel parameters.

Both commands change the live kernel setting in the /proc filesystem. After running either of those commands, you should run the sysctl vm.swappiness command to verify that the kernel setting has changed.

But those commands only change the swappiness value for the currently running system. A reboot returns the value to its default. I needed to ensure that this change is made persistent across reboots.

But first, the failure

To permanently change the kernel vm.swappiness variable, I used the procedure described in my previous article, How I disabled IPv6 on Linux, to add the following line to the end of the /etc/default/grub file:

GRUB_CMDLINE_LINUX="vm.swappiness=1"

I then ran the grub2-mkconfig command as root to rebuild the /boot/grub2/grub.cfg file. However, testing with VMs and real hardware showed that it did not work, and the swappiness value did not change. So I tried another approach.

And the success

Between this failure at startup time, the one I describe in the How I disabled IPv6 on Linux article, and other startup issues I explored due to encountering those two, I decided that this was a Linux startup timing problem. In other words, some required services, one of which might be the network itself, were not up and running, which prevented these kernel option changes from being committed to the /proc filesystem, or they were committed and then overwritten when the service started.

I could make all of these work as they should by adding them to a new file, /etc/sysctl.d/local-sysctl.conf with the following content, which includes all of my local kernel option changes:

###############################################
#            local-sysctl.conf                #
#                                             #
# Local kernel option settings.               #
# Install this file in the /etc/sysctl.d      #
# directory.                                  #
#                                             #
# Use the command:                            #
# sysctl -p /etc/sysctl.d/local-sysctl.conf   #
# to activate.                                #
#                                             #
###############################################
###############################################
# Local Network settings                      #
# Specifically to disable IPV6                #
###############################################
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1

###############################################
# Virtual Memory                              #
###############################################
# Set swappiness
vm.swappiness = 1

I then ran the following command, which activated only the kernel options in the specified file:

# sysctl -p /etc/sysctl.d/local-sysctl.conf
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
vm.swappiness = 13

This is a more targeted approach to setting kernel options than I used in my article about disabling IPv6.

Reporting the bug

At the time of this writing, there is no true fix for the root cause of this problem–whatever the cause. There is a way to temporarily circumvent the issue until a fix is provided. I used the /etc/sysctl.d/local-sysctl.conf file that I had created for testing and added a systemd service to run at the end of the startup sequence, wait for a few seconds, and run sysctl on that new file. The details of how to do that are in the How I disabled IPv6 on Linux article.

I had already reported this as bug 2103517 using Red Hat's Bugzilla when trying to disable IPv6. I added this new information to that bug to ensure that my latest findings were available to the kernel developers.

You can follow the link to view the bug report. You do not need an account to view bug reports.

Final thoughts

After experimenting to see how well I could reproduce the symptoms, along with many others, I have determined that the vm.swappiness setting of 60 is far too aggressive for many large-memory Linux systems. Without a lot more data points than those of my own computers, all I can tentatively conclude is that systems with huge amounts of RAM that get used only infrequently are the primary victims of this problem.

The immediate solution to the problem of local kernel option settings not working is to set them after startup. The automation I implemented is a good example of how to use systemd to replace the old SystemV startup file rc.local.

This bug had not been previously reported. It took a few days of experimenting to verify that the general problem in which locally-set kernel options were not being set or retained at startup time was easily repeatable on multiple physical and virtual systems. At that point, I felt it important to report the bug to ensure it gets fixed. Reporting it is another way I can give back to the Linux community.

I recently experienced another interesting problem in the Linux startup sequence that has a circumvention–not a solution. It started quite unexpectedly.

Image by:

opensource.com

Linux Sysadmin What to read next How I recovered my Linux system using a Live USB device This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

Open source blockchain development: Get started with Hyperledger FireFly

Tuesday 13th of September 2022 07:00:00 AM
Open source blockchain development: Get started with Hyperledger FireFly Nicko Guyer Tue, 09/13/2022 - 03:00

It takes more than a blockchain node to build enterprise-grade applications at scale. As a result, developers often find themselves building plumbing from scratch to make their business logic work. The release of Hyperledger FireFly changed blockchain development, offering developers a full stack of tools to build and scale secure web applications using familiar APIs. FireFly’s next-gen platform simplifies development, making it easy to connect across multiple public and private chains while running many use cases simultaneously. Whether you want to build on permissioned chains like Hyperledger Fabric, Corda, or Enterprise Ethereum, or public chains like Ethereum, Polygon, Avalanche, Optimism, BNB Chain, Arbitrum, Moonbeam, or Fantom, FireFly has you covered.

In this article I'll walk you through where to download Hyperledger FireFly, how to set up a local development environment, and introduce you to the FireFly Sandbox. But first, a quick introduction to the Supernode.

What is a Supernode?

Hyperledger FireFly is an open source project that was contributed to the Hyperledger Foundation by Kaleido, a blockchain and digital asset platform provider. To make FireFly a reality, the Raleigh, NC-based company collaborated with the blockchain community to fit vital technology components into an enterprise-grade, pluggable development and runtime stack called a Supernode.

Image by:

(Nicko Guyer, CC BY-SA 4.0)

This development stack offers three key advantages to blockchain developers, especially those looking to build enterprise-grade applications with scale.

  • Accelerate: Hyperledger FireFly helps developers create applications on the blockchain protocol of their choice, and build quickly with familiar REST APIs. Users can leverage pre-built services for tokens, wallets, storage, and identity to reach production faster.
  • Orchestrate: Hyperledger FireFly makes it easier to manage data end-to-end from blockchain to back-office. APIs allow developers to trigger business processes based on blockchain activities, and off-chain storage and messaging to protect sensitive data.
  • Support: Hyperledger FireFly supports high-volume workloads, integrates with existing IT systems and databases, and communicates with network participants.
Getting Started with Hyperledger

FireFly makes it super easy to build powerful blockchain applications. Installing the stack on your machine is a simple process, too. Below I'm going to walk you through the three step process to get up and running so you can start testing the FireFly functionality today.

Image by:

(Nicko Guyer, CC BY-SA 4.0)

Install FireFly CLI

The FireFly command-line interface (CLI) creates local FireFly stacks for offline development of blockchain apps. Having FireFly locally allows developers to test and iterate on ideas without worrying about setting up extra infrastructure.

The easiest way to install the FireFly CLI is to download a pre-compiled binary of the latest release. To do this, visit the release page.

Next, extract the binary and move it to /usr/bin/local. Assuming you downloaded the package from GitHub into your Downloads directory:

$ sudo tar -zxf ~/Downloads/firefly-cli_*.tar.gz -C /usr/local/bin ff

This places the ff executable in /usr/local/bin.

If you downloaded the package from GitHub to a different directory, change the tar command above to wherever the firefly-cli_*.tar.gz file is located.

Alternatively, you can install the FireFly CLI using Go. If you have a local Go development environment, and you have included ${GOPATH}/bin in your path, you can use Go to install the FireFly CLI by running:

$ go install github.com/hyperledger/firefly-cli/ff@latest

Finally, verify the installation by running ff version. This prints the current version:

{ "Version": "v1.1.0", "License": "Apache-2.0" }

With the FireFly CLI installed, you are ready to run some Supernodes on your machine.

More great content Free online course: RHEL technical overview Learn advanced Linux commands Download cheat sheets Find an open source alternative Explore open source resources Start Your Environment

A FireFly stack is a collection of Supernodes that work together on a single development machine. A stack has multiple members (also referred to as organizations). Every member has their own Supernode within the stack. This allows developers to build and test data flows with a mix of public and private data between various parties, all within a single development environment.

Image by:

(Nicko Guyer, CC BY-SA 4.0)

Creating a new FireFly stack is relatively easy. The ff init command creates a new stack for you, and prompts you for a few details such as the name, and how many members you want in your stack.

There are also some settings you can change. The defaults are the simplest way to get going, but you can see a full list of options by running ff init --help.

Once you've created your stack, use the command ff start dev to run your environment.

After your stack has started, it prints the links to each member's UI, and the Sandbox for that node.

Use the FireFly Sandbox Image by:

(Nicko Guyer, CC BY-SA 4.0)

Each member gets an instance of the FireFly Sandbox as well. The Sandbox is like an example application. It can be used to test, iterate, and practice using FireFly features. It provides code snippets as examples of how to build those features into your own application backend.

There are a couple of things in the Sandbox you may want to check out to experience the full capabilities of Hyperledger FireFly.

The Messages tab helps you send messages, and view the data payload, in every member's FireFly Explorer, or send a private message to one member, and verify that the data payload is not visible in a third member's FireFly Explorer. You can send an image file, and download it from another member's FireFly Explorer.

The Tokens tab creates a non-fungible token pool, and allows you to transfer an NFT to another member and verify the account balances in FireFly Explorer.

The Contracts tab can create a contract interface and API, then view the Swagger UI for your new API, or create an event listener. You can also use the Swagger UI to call a smart contract function that emits an event. Any event received in the Sandbox also shows up in FireFly Explorer.

Build your app

Hyperledger FireFly brings a complete open source stack for developers who want to build and scale secure, enterprise-grade applications with access to blockchain technology. It's simple to install on your machine, and the Sandbox allows developers to view code snippets and test ideas—all so blockchain applications reach production faster. Read more about Hyperledger FireFly's capabilities in the project documentation and give it a try yourself.

Hyperledger FireFly brings a complete open source stack for developers who want to build and scale secure, enterprise-grade applications with access to blockchain technology.

Image by:

Opensource.com

Blockchain What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. Register or Login to post a comment.

Remixing Linux for blind and visually impaired users

Tuesday 13th of September 2022 07:00:00 AM
Remixing Linux for blind and visually impaired users Vojtech Polasek Tue, 09/13/2022 - 03:00

When I was around 5 years old, my father brought home our first computer. From that moment on, I knew I wanted to pursue a career in computers. I haven't stopped hanging around them since. During high school, when considering which specific area I wanted to focus on, I started experimenting with hacking, and that was the moment I decided to pursue a career as a security engineer.

I'm now a software engineer on the security compliance team. I've been at Red Hat for over two years, and I work remotely in the Czech Republic. I've used Linux for about 12 years, mainly Arch Linux and Fedora, but I've also administered Debian, Gentoo, and Ubuntu in the past.

Image by:

(Vojtech Polasek, CC BY-SA 4.0)

Photo description: Black and white image of a smiling Vojtech, with a red frame around it and an illustrated paper airplane in the background.

Outside of my day job, I play blind football, and I'm involved in various projects connecting visually impaired and sighted people together, including working in a small NGO that runs activities for blind and visually impaired people. I'm also working on an accessible Fedora project currently called Fegora, an unofficial Linux distribution aimed at visually impaired users.

The assistive technology stack

When I use a smart device, I need several pieces of assistive technology. The first and most essential is called a screen reader. This is software that presents what's on the screen to blind or visually impaired people, either through speech or through braille (basically, it tries to serve as our eyes). It can read out notifications and tell me which button or page element I'm focusing on, allowing me to interact with graphical user interfaces.

Screen readers use speech synthesis to speak aloud what appears on the screen. There are a variety of speech synthesizers, and some voices are more "natural-sounding" than others. The one I use, Espeak, is not very natural-sounding, but it's lightweight and fast. It also supports almost all languages, including Czech (which I use).

Finally, I use a Braille display, a device that represents a line of text in Braille. I use this a lot, especially when I'm coding or doing code reviews. It's easier to grasp the structure of code when I can freely move from one code element to another by touch. I can also use its buttons to move the cursor to the character or area of the screen I'm interested in, and it has a Braille keyboard too if I want to use it.

More Linux resources Linux commands cheat sheet Advanced Linux commands cheat sheet Free online course: RHEL technical overview Linux networking cheat sheet SELinux cheat sheet Linux common commands cheat sheet What are Linux containers? Our latest Linux articles How I use assistive technology on a daily basis

When using a computer as a blind or visually impaired person, there are a couple of things that are relatively straightforward to do using the tech above. Personally, these are a few of the things I do every day:

  • The text console is pretty much my favorite application. As a general rule, when something's in text, then blind people can read it with a screen reader (this doesn't hold true in all cases, but in most.) I mainly use the console for system management, text editing, and working with guidance and documentation.
  • I browse the web and interact with websites.
  • I code and do code reviews using VSCode and Eclipse.
  • I send emails and instant messages.
  • I can use word processing software, like Google Docs (which is not open source, but common in the modern office) and LibreOffice. Google Docs developers have added a lot of keyboard shortcuts, which I can use to move around documents, jump to headings or into comments, and so on.
  • I can play multimedia, usually. It depends on how the application is written. Some media players are more accessible than others.
Possible but painful

This brings me to tasks that aren't so easy. I like to call these "possible but painful".

PDF files can be difficult. Sometimes I end up needing to use optical character recognition (OCR) software to convert images to text. For example, recently I needed to read a menu for a restaurant. They had the PDF of their menu on their website, but it had been flattened, and didn't have a text layer. For me, this shows up as a blank screen. I had to use an OCR application from my smartphone to extract the text for me. Not only is this an extra step, but the resulting "translation" of the text isn't always entirely accurate.

Viewing and creating a presentation can be problematic. To work around this, I create slides in HTML, using software such as Pandoc, which can process markdown and convert it into slides. I've been using this for many years and it works well—it allows me total control of the resulting slides, because the markdown is just simple text.

Video games can be made more accessible by basing them on sound or text. However, playing games can be doubly challenging on Linux as not only would you need to find an accessible game, but most PC games are also native to Windows so you would be dealing with some compatibility issues as well.

Some websites and interfaces are more difficult to navigate than others. These issues are often quite easy to solve just by setting some attributes correctly. In general, lots of web content comes in the form of images, especially today. One of the easiest ways to make web content more accessible is to make sure that alternative text is added to images so that screen readers can read it out, and people who cannot distinguish the image have some idea what's there. Another thing I experience a lot is unlabeled controls: you know there's a button or a check box but you don't know what it does.

The Fegora project optimises Linux for accessibility

Developers don't intentionally set out to build applications that aren't accessible. The problem is that they usually don't know how to test them. There aren't many blind Linux users, so there aren't many people testing the accessibility of applications and providing feedback. Therefore, developers don't produce accessible applications, and they don't get many users. And so the cycle continues.

This is one thing we hope to tackle with the Fegora project. We want to create a Fedora remix that's user-friendly for visually impaired and blind users. We hope it will attract more users, and that those users start discovering issues to report, which will hopefully be solved by other developers in the open source community.

So why are we doing this? Well, it's important to point out that Fedora is not an inaccessible distribution by design. It does have many accessibility tools available in the form of packages. But these aren't always present from the beginning, and there are a lot of small things which need to be configured before it can be proficiently used. This is something that can be discouraging to a beginner Fedora user.

We want Fegora to be as friendly and predictable for a blind user as possible. When a user launches a live image, the screen immediately starts being read as soon as a graphical user interface appears. All environment variables needed for accessibility are loaded and configured correctly.

Fegora brings the following changes, among others:

  • Environment variables for accessibility are configured from the start.
  • The Orca screen reader starts as soon as the graphical interface loads.
  • A custom repo is added with extra voice synthesis and packaged software.
  • Many alternative keyboard shortcuts have been added.
  • There's a special script that can turn your monitor on and off. Many users do not need the monitor at all and having it off is a great power saver!
So how can you help?

First, if you'd like to contribute to Fegora (or just spread the word), you can find out more on our repository.

Additionally, when working on a team with someone who has a visual impairment, there might be some additional considerations depending on the accessibility tech being used. For example, it's not easy for us to listen to someone and read at the same time, because we are basically getting both things through audio, unless someone is very proficient with the Braille display.

Lastly, bear in mind that blind and visually impaired users consume the same end products as you do, whether that's presentation slides or websites or PDFs. When building products or creating content, your choices have a huge effect on accessibility and how easy it is for us to engage with the end result. Know that we are here, we love to use computers and technology, and we're often willing to help you test it, too.

Image by:

(Vojtech Polasek, CC BY-SA 4.0)

 Image description: Vojtech holding a football. He is wearing a football uniform and protective goggles.

Fegora, a Fedora project, is an unofficial Linux distribution aimed at visually impaired users.

Image by:

Opensource.com

Linux Accessibility Diversity and inclusion What to read next Use this open source screen reader on Windows Usability and accessibility start with open communication This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. 1 Comment Register or Login to post a comment. Peter Cheer | September 17, 2022 Register or Login to like

This sounds like an interesting project Vojtech, there is also the Ubuntu derived distro Vinux (https://vinux.org.uk/) for Linux users with visual impairments.

How I recovered my Linux system using a Live USB device

Monday 12th of September 2022 07:00:00 AM
How I recovered my Linux system using a Live USB device David Both Mon, 09/12/2022 - 03:00

I have a dozen or so physical computers in my home lab and even more VMs. I use most of these systems for testing and experimentation. I frequently write about using automation to make sysadmin tasks easier. I have also written in multiple places that I learn more from my own mistakes than I do in almost any other way.

I have learned a lot during the last couple of weeks.

I created a major problem for myself. Having been a sysadmin for years and written hundreds of articles and five books about Linux, I really should have known better. Then again, we all make mistakes, which is an important lesson: You're never too experienced to make a mistake.

I'm not going to discuss the details of my error. It's enough to tell you that it was a mistake and that I should have put a lot more thought into what I was doing before I did it. Besides, the details aren't really the point. Experience can't save you from every mistake you're going to make, but it can help you in recovery. And that's literally what this article is about: Using a Live USB distribution to boot and enter a recovery mode.

The problem

First, I created the problem, which was essentially a bad configuration for the /etc/default/grub file. Next, I used Ansible to distribute the misconfigured file to all my physical computers and run grub2-mkconfig. All 12 of them. Really, really fast.

All but two failed to boot. They crashed during the very early stages of Linux startup with various errors indicating that the /root filesystem could not be located.

I could use the root password to get into "maintenance" mode, but without /root mounted, it was impossible to access even the simplest tools. Booting directly to the recovery kernel did not work either. The systems were truly broken.

More Linux resources Linux commands cheat sheet Advanced Linux commands cheat sheet Free online course: RHEL technical overview Linux networking cheat sheet SELinux cheat sheet Linux common commands cheat sheet What are Linux containers? Our latest Linux articles Recovery mode with Fedora

The only way to resolve this problem was to find a way to get into recovery mode. When all else fails, Fedora provides a really cool tool: The same Live USB thumb drive used to install new instances of Fedora.

After setting the BIOS to boot from the Live USB device, I booted into the Fedora 36 Xfce live user desktop. I opened two terminal sessions next to each other on the desktop and switched to root privilege in both.

I ran lsblk in one for reference. I used the results to identify the / root partition and the boot and efi partitions. I used one of my VMs, as seen below. There is no efi partition in this case because this VM does not use UEFI.

# lsblk
NAME          MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
loop0           7:0    0  1.5G  1 loop
loop1           7:1    0    6G  1 loop
├─live-rw     253:0    0    6G  0 dm   /
└─live-base   253:1    0    6G  1 dm  
loop2           7:2    0   32G  0 loop
└─live-rw     253:0    0    6G  0 dm   /
sda             8:0    0  120G  0 disk
├─sda1          8:1    0    1G  0 part
└─sda2          8:2    0  119G  0 part
  ├─vg01-swap 253:2    0    4G  0 lvm  
  ├─vg01-tmp  253:3    0   10G  0 lvm  
  ├─vg01-var  253:4    0   20G  0 lvm  
  ├─vg01-home 253:5    0    5G  0 lvm  
  ├─vg01-usr  253:6    0   20G  0 lvm  
  └─vg01-root 253:7    0    5G  0 lvm  
sr0            11:0    1  1.6G  0 rom  /run/initramfs/live
zram0         252:0    0    8G  0 disk [SWAP]

The /dev/sda1 partition is easily identifiable as /boot, and the root partition is pretty obvious as well.

In the other terminal session, I performed a series of steps to recover my systems. The specific volume group names and device partitions such as /dev/sda1 will differ for your systems. The commands shown here are specific to my situation.

The objective is to boot and get through startup using the Live USB, then mount only the necessary filesystems in an image directory and run the chroot command to run Linux in the chrooted image directory. This approach bypasses the damaged GRUB (or other) configuration files. However, it provides a complete running system with all the original filesystems mounted for recovery, both as the source of the tools required and the target of the changes to be made.

Here are the steps and related commands:

1. Create the directory /mnt/sysimage to provide a location for the chroot directory.

2. Mount the root partition on /mnt/sysimage:

# mount /dev/mapper/vg01-root /mnt/sysimage

3. Make /mnt/sysimage your working directory:

# cd /mnt/sysimage

4. Mount the /boot and /boot/efi filesystems.

5. Mount the other main filesystems. Filesystems like /home and /tmp are not needed for this procedure:

# mount /dev/mapper/vg01-usr usr

# mount /dev/mapper/vg01-var var

6. Mount important but already mounted filesystems that must be shared between the chrooted system and the original Live system, which is still out there and running:

# mount --bind /sys sys

# mount --bind /proc proc

7. Be sure to do the /dev directory last, or the other filesystems won't mount:

# mount --bind /dev dev

8. Chroot the system image:

# chroot /mnt/sysimage

The system is now ready for whatever you need to do to recover it to a working state. However, one time I was able to run my server for several days in this state until I could research and test real fixes. I don't really recommend that, but it can be an option in a dire emergency when things just need to get up and running–now!

The solution

The fix was easy once I got each system into recovery mode. Because my systems now worked just as if they had booted successfully, I simply made the necessary changes to /etc/default/grub and /etc/fstab and ran the grub2-mkconfig > boot/grub2/grub.cfg command. I used the exit command to exit from chroot and then rebooted the host.

Of course, I could not automate the recovery from my mishap. I had to perform this entire process manually on each host—a fitting bit of karmic retribution for using automation to quickly and easily propagate my own errors.

Lessons learned

Despite their usefulness, I used to hate the "Lessons Learned" sessions we would have at some of my sysadmin jobs, but it does appear that I need to remind myself of a few things. So here are my "Lessons Learned" from this self-inflicted fiasco.

First, the ten systems that failed to boot used a different volume group naming scheme, and my new GRUB configuration failed to consider that. I just ignored the fact that they might possibly be different.

  • Think it through completely.
  • Not all systems are alike.
  • Test everything.
  • Verify everything.
  • Never make assumptions.

Everything now works fine. Hopefully, I am a little bit smarter, too.

The Fedora Live USB distribution provides an effective solution to boot and enter a recovery mode.

Image by:

Photo by Markus Winkler on Unsplash

Linux Sysadmin What to read next This work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License. 1 Comment Register or Login to post a comment. Hasan | September 12, 2022 Register or Login to like

Thank you Mr David for this nice article.

More in Tux Machines

today's howtos

  • How to install go1.19beta on Ubuntu 22.04 – NextGenTips

    In this tutorial, we are going to explore how to install go on Ubuntu 22.04 Golang is an open-source programming language that is easy to learn and use. It is built-in concurrency and has a robust standard library. It is reliable, builds fast, and efficient software that scales fast. Its concurrency mechanisms make it easy to write programs that get the most out of multicore and networked machines, while its novel-type systems enable flexible and modular program constructions. Go compiles quickly to machine code and has the convenience of garbage collection and the power of run-time reflection. In this guide, we are going to learn how to install golang 1.19beta on Ubuntu 22.04. Go 1.19beta1 is not yet released. There is so much work in progress with all the documentation.

  • molecule test: failed to connect to bus in systemd container - openQA bites

    Ansible Molecule is a project to help you test your ansible roles. I’m using molecule for automatically testing the ansible roles of geekoops.

  • How To Install MongoDB on AlmaLinux 9 - idroot

    In this tutorial, we will show you how to install MongoDB on AlmaLinux 9. For those of you who didn’t know, MongoDB is a high-performance, highly scalable document-oriented NoSQL database. Unlike in SQL databases where data is stored in rows and columns inside tables, in MongoDB, data is structured in JSON-like format inside records which are referred to as documents. The open-source attribute of MongoDB as a database software makes it an ideal candidate for almost any database-related project. This article assumes you have at least basic knowledge of Linux, know how to use the shell, and most importantly, you host your site on your own VPS. The installation is quite simple and assumes you are running in the root account, if not you may need to add ‘sudo‘ to the commands to get root privileges. I will show you the step-by-step installation of the MongoDB NoSQL database on AlmaLinux 9. You can follow the same instructions for CentOS and Rocky Linux.

  • An introduction (and how-to) to Plugin Loader for the Steam Deck. - Invidious
  • Self-host a Ghost Blog With Traefik

    Ghost is a very popular open-source content management system. Started as an alternative to WordPress and it went on to become an alternative to Substack by focusing on membership and newsletter. The creators of Ghost offer managed Pro hosting but it may not fit everyone's budget. Alternatively, you can self-host it on your own cloud servers. On Linux handbook, we already have a guide on deploying Ghost with Docker in a reverse proxy setup. Instead of Ngnix reverse proxy, you can also use another software called Traefik with Docker. It is a popular open-source cloud-native application proxy, API Gateway, Edge-router, and more. I use Traefik to secure my websites using an SSL certificate obtained from Let's Encrypt. Once deployed, Traefik can automatically manage your certificates and their renewals. In this tutorial, I'll share the necessary steps for deploying a Ghost blog with Docker and Traefik.

Red Hat Hires a Blind Software Engineer to Improve Accessibility on Linux Desktop

Accessibility on a Linux desktop is not one of the strongest points to highlight. However, GNOME, one of the best desktop environments, has managed to do better comparatively (I think). In a blog post by Christian Fredrik Schaller (Director for Desktop/Graphics, Red Hat), he mentions that they are making serious efforts to improve accessibility. Starting with Red Hat hiring Lukas Tyrychtr, who is a blind software engineer to lead the effort in improving Red Hat Enterprise Linux, and Fedora Workstation in terms of accessibility. Read more

Today in Techrights

Android Leftovers