Category: Technical Zone

Why is measuring performance with the CPU a mistake?

Reading Time: < 1 minute

In his Upscale 2018 presentation last march, Brendan Gregg, senior performance architect at Netflix, explains why the CPU isn’t a good metric for measuring performance.

Brendan Gregg, senior performance architect at Netflix”
Everyone uses %CPU to measure performance, but everyone is wrong

This is an opportunity for us to revisit what we have been pushing for a long time: How to measure performance and resource consumption reliably?

Brendan Gregg explains that the CPU rate can indicate false information. For instance, a busy CPU doesn’t necessarily indicate that the CPU is busy doing calculations but is waiting for information. It’s therefore necessary to use more advanced tools to analyze what happens in the lower layers of the code. Brendan Gregg even concludes that this problem will get worse with the acceleration of processor speed.

We have observed this phenomenon through our R&D. Indeed, a high CPU rate doesn’t necessarily mean that the CPU is requested and conversely a low CPU could hide overconsumption from another cache.

The problem is even more present in mobility. Indeed, the current platforms are heterogeneous: CPU, GPU, Radio, multiple interfaces… It’s therefore complex to follow the current performance of these platforms and it’s necessary to use various and complex profiling tools.

Our approach at Greenspector is different, we propose to use energy as a more reliable metric. Any component solicited consumes energy, monitoring energy consumption will detect over-consumption of resources.

For more information, we invite you to read our article Why you should measure the energy consumption of your software?.

How to analyze the behavior of my smartphone with Battery Historian?

Reading Time: 3 minutes

At GREENSPECTOR, our R&D teams have been working for several years on the energy measurement of smartphones. Several years of research and innovation that allow us today to offer a unique product to simply measure the energy data of smartphones. However, there are tools that can complete the analysis of the behavior of the battery and the phone. Battery Historian is one of them.

Battery Historian is a tool developed by Google, launched in 2016, which allows one to analyze the behavior of a phone and more precisely to review information and events related to the battery. This is, we will see, an expert analysis tool. Several metrics and insights are available: radio cells, communication… all this correlates with the battery level.

Battery Historian usage tutorial

1) First of all, install Docker. Docker is free software that automates the deployment of applications in software containers. Personally, I am on Ubuntu so I use the quick install scripts:

curl -fsSL get.docker.com -o get-docker.sh
sudo sh get-docker.sh

2) From your command line interface, launch the Docker image:

docker run -d -p 9999:9999 bhaavan/battery-historian

3) You can now access Battery Historian from localhost:9999

You must now retrieve the detailed information of your phone’s system. For this, you must have previously installed the Android SDK and activated the developer mode on your phone.

4) Plug your phone into USB to your PC.

5) From your command line interface, retrieve the phone’s system information file with the following command:

adb bugreport bugreport.zip

6) You can now upload the file bugreport.txt in Battery Historian

You will then get your analysis report:

You can now observe the battery level on the right axis (from 0 to 100%), this level is represented by the black curve. On the left axis and the other lines, you will find all other information about the metrics and insights reported by the system.

Below this graph, you will find statistics of all information about specific applications or system information of your phone. Allowing you an overview of what is happening with your device. For example, you can analyze the impact of a particular application on the battery during use

Our opinion on Battery historian

Battery Historian is a powerful tool that allows you to analyze the behavior of the battery and most importantly understand what is happening in the system. But it is also a complex tool. There is a lot of information and it quickly becomes difficult to find the causes of the discharge. In addition, since the tool is based on the battery level, it is necessary to allow the application you want to analyze to run for a while. It is therefore a useful tool for a detailed analysis of the system but to put in expert hands.

Why should you care about your application’s impact on the end user’s battery life?

Reading Time: 5 minutes

TL ; DR

Smartphones’ battery life has been plateauing for a few years. Manufacturers offer batteries with a greater capacity, users state battery life is one of the main purchasing criteria, and, yet, the average battery life of devices doesn’t increase. Why that, you may ask? Because hardware itself is more powerful, thus more energy-intensive; and also because applications consume more and more resources.

As a consequence, in order to improve the way users perceive their devices, manufacturers point at the most energy-intensive applications, encouraging developers to be more virtuous in terms of development. Google has the same reasoning with the Play Store, favoring the ranking of the lightest applications. So, yes, even a “general public” application used only from time to time must be optimized to consume the least battery possible. By avoiding that your application appears as “energy-intensive”, you improve its indexing, retention and in fine the user experience. Many factors key to the success of your app.

Battery life, hardware selection criterion

Any smartphone user has already faced a critical situation because of a very low battery level: no way to be reached anymore, an address impossible to found right before an important meeting, no train or plane ticket to show the controller… The – almost – vital need to maintain a fully charged battery has become a true concern for half the world population.

Hence, it isn’t surprising that battery life is one of the main selection criteria for buyers. Manufacturers are constantly working on improving the efficiency of batteries and hardware. And the technology press follows closely the subject.
While waiting for a hypothetical new and ideal technology, solutions regularly emerge. The range of applicative “solutions” allowing the optimization of battery consumption is very wide on the different stores, however with no strong impact on battery life improvement. Facing this, users fall back on palliatives: USB battery extensions (“power banks”), mini solar panels,… many accessories adding up to the initial budget and cluttering up bags.

In spite of all that, the « low battery anxiety » phenomenon keeps rising.

Aren’t manufacturers doing something?

This issue is well known from manufacturers. Indeed, improving battery life is a true challenge operating at the level of every single component: of course batteries, but also size and screen technology, choice of electronic components, their complete integration, and software layers. Manufacturers integrate OS (like Android), develop drivers for electronic components but also develop applications for users, by creating them or adapting AOSP Android applications.

But why working on applications?

Well, simply because the hardware platform is not the only cause of battery discharge. Software layers are not innocent in this issue, far from it actually. If we think about it a bit more, it is normal! Here is an example of a simplified curve representing the energy consumption during the loading of an application:


We can see the overconsumption caused by the launch of the application.

As you might know already, all applications don’t have the same impact on the phone’s overall battery life. Here is the comparison of consumption when launching an e-commerce application, which basically doubles:

The use of features such as GPS, Bluetooth or video (or in the near future AR or VR features) has an impact, but it is not the only cause of inequalities. Indeed, depending on the design, use, external services, sensors it will solicit… an application will consume more or less.

Unfortunately (or fortunately), applications do have a strong impact on our smartphones’ battery life.

Applications’ impact, a major criterion

Manufacturers are aware of this applications’ impact. They work on optimizing their own applications, but have no power on applications installed on the phone once it is out of the factory… It starts with phone operators or integrators, who add their own applications to the device. Then users install whatever they want. All these applications will have a more or less important impact on battery life. In the end, even if the manufacturer isn’t directly responsible, the user will associate the poor battery life of the device with the brand and the phone’s model: 71% of users consider that the quality of an application can have influence on the brand image (Source: baromètre des usages mobiles ,Juin 2017, EBG).

As a consequence, it is in the manufacturers’ best interest to encourage applications editors to control their apps’ battery consumption. In the same way, it is in Google’s interest to promote virtuous applications on the Play Store, so that it reflects a positive image of the Android ecosystem, in comparison with Apple’s… and vice versa.

What are manufacturers doing? They report you to users.

Manufacturers should find interests in sensitizing users to this complex battery life issue and making the applications designer take the fall.

For example, let’s cite Huawei which now sends out explicit notifications to its users in case of a detected overconsumption:

This same will to promote energy-thrifty applications is now integrated in Google and Apple ecosystems. It has been a while now that platforms don’t hesitate pointing at energy-intensive applications. In order to do that, the system lists each application’s share of consumption:

Android 8.10 Oreo latest version reinforces this functionality by adding a notification of consuming applications :

But this reasoning isn’t just limited to the user’s phone as we can also find it on the stores. That way, Google takes into account consumed energy in its ASO indexing criterion for applications.
You probably know by now that efficiency is key for a good SEO, and Google confirms it also is for applications (ASO). It is a criterion you cannot neglect since 80% of users use the store as entry point for their application search (Source: baromètre des usages mobiles, Juin 2017,EBG).

Want an example? Google explains what Busun went through, which is going from a 4.1 to a 4.5 rating by improving its performance.

What does the user do?

Let’s suppose the user found your application on the store, was convince to install it, well now remains the hard part: retention of the app. Either your app belongs to the dozen of “critical” must-have applications with no real equivalent (Google Maps, Facebook…): in that case the user accepts the battery consumption as necessary, he/she complains but will keep your app.

Or, your application belongs to the millions of other apps, considered non-essential – or with numerous alternatives (online banks, informative press, etc): in this case, the user will be quick to uninstall your app to switch to your competitors’. And he/she may even leave a bad rating or mean comment to your app on the store.

Conclusion

General public applications owners sometimes think the impact of their app on the device’s battery life isn’t an issue. After all, it is only used from time to time; it is not necessarily that ONE application that is discharging the whole battery, right? Well, we saw it isn’t true. Even a general public application can strongly be impacted by a behavioral default. And users WILL realize it: degraded indexing, low retention, negative comments… The success of an application also comes from the good management of its efficiency.

Android Memory: the Ultimate Metric Guide

Reading Time: 5 minutes

Because we care about your applications efficiency, we introduce you to the ultimate metric guide to manage your application’s memory on Android devices.

Memory syllabus

Pages: Blocks used to retrieve data from disk to memory. Main part of virtual memory management. Most of the time page size is 4kb.

Private / Shared memory: Private memory is composed of pages that are only used by the process. Shared memory is composed of pages which are used by other processes. It is possible that a page goes from private to shared if another process references this page.

Clean / Dirty memory: Dirty pages are pages that are written out to the disk. Hence, clean pages are pages that have been read.

VSS (Virtual Set Size): It is the total number of pages accessible by a process. It includes both shared and private pages, but also other memory space like malloc. Not used.

RSS (Resident Set Size): Total of all shared libraries. No information on how many processes use the page. Thus, not a very useful information (PSS is more important).

Pss (proportional set size): It is the count of pages a process has in memory, in which private pages are added to shared pages and divided by the number of processed that share it. If a process has 2 private pages and 6 private pages used by 2 others processes, the pss in number of pages is 2+6/3 so 4.

Uss (Unique set size): The number of unique pages associated to a process, meaning private pages! Generaly VSS > RSS > PSS > USS

Java / native memory: Java memory is the memory managed by the Android System (Dalvik or ART). Typically your Java or Kotlin objects! Native memory is usually generated object. Even if you have no C/C++ application, the Android framework uses some native method.

ZRam: Android has no swap partition, Zram is RAM but compression is used to store unnecessary data to RAM. Swapped dirty in meminfo seems to be equivalent.

Stack / heap: Heap is the object memory space. Stack is for local variable. As a result, objects are created in Heap. Every thread has a stack space.
Allocated Heap Size : Android is a multitasking system, in order to save memory for other processes, it allocates a memory size in the overall memory. This size depends on the smartphone. If your application needs more memory, you will have a typical “out of memory” error.

MMAP: System calls that map file to memory. In Android you have mmap for each type of file (Dex, APK, ART…). It can be named “Code” in some tools.

Graphics Buffer: Memory used to store graphical objects.

User/Kernal memory: Process runs in user space. Kernel space is dedicated to Android system calls.

Memory organization

Android Memory is organized with virtual addresses. Mapping is the following (Ascending address):

  • User Space
    • Data space (text, data, instructions…)
    • Heap
    • libs (Shared libraries, .jar…)
    • Stack
  • Kernel Space

The memory used by your application is in the user space (mostly in heap and stack).

Tools

Android system parameter: The information in the UI of Android gives two information : maximum consumption and average information (during 3 hours). The average is complicated to analyze and the maximum consumption is a first indication for the user.

Top: RSS and VSS are used, as a consequence information of Top are not so meaningful.

Memory Profiler in Android Studio (>3.0): Private memory only is measured and is divided in domains:

  • Native / Java
  • Code
  • Graphics Buffer

Android Monitor in Android Studio (<3.0): Private memory only is measured. Less domains are measured than in Android Memory > 3

Dumpsys: Dumpsys measure the PSS memory (both private and shared) and also SWAP dirty

Android MemInfo API: PSS memory is used (private and shared)

/proc/meminfo: Global memory of the platform, no information about your process

Which metrics must be used?

USS/private memory: This memory is dedicated to your process. Thus it is interesting to analyze precisely the object dedicated to your application. Also, you can detect a memory leak, for example, if you kill your process, the dedicated memory must be zero. Nonetheless, your application used libraries and other shared codes so you won’t see memory that can be shared with other processes.

PSS/Private and shared memory: It is more complicated to analyze this memory because it integrates the memory of other processes (with a correction factor). However, during your measurement, if you close other apps running in the background, this impact will be minimized.

VSS/RSS: Too many memory pages taken into account so this metric is not interesting to analyze.

How to test memory?

Static: You just have to run your application and launch the memory tools. It permits to get an overview of the memory consumption of your application. You can use this technique to reduce the overall memory used: code, java heap…

Long Running: You let the tools run and you observe the evolution of memory. That way, you can identify a potential memory leak: the memory can increase regularly. If there is a garbage collector which reduces regularly the memory down to the initial value it is OK, if not, there is a leak.

Switching between activities: You can go between pages and see if the memory is released every time. You can do that with monkey tests.

Rotating screen: If you rotate your screen, the activity of your application will potentially create a memory leak.

Put application in the background: If your application is in the background, some memory will be released (for example graphics). You can release more memory (object, library…).

How to reduce memory?

Optimize Image: Images are major causes of memory consumption. You can use Glide library to help you manage it or optimize the management with Android API

Release Memory when Android asks for more memory : As the system is low on memory, an event is sent to all applications to release memory. Override onTrimMemory() to release memory in this case

Use specific containers: Generic collections like hashmap are not dedicated to Android, some containers are more adapted, such as SparseArray

Reduce the size of your APK: The size of your apk (especially ressources and libraries) could increase the memory consumption

Reduce the number of threads: Android provides services to make small tasks and to avoid memory consumption, like AsyncTask

Favor static inner classes over non-static: Non-static inner classes create extraneous references to out activity. This is a cause of memory leak. You should choose static.

FAQ

How to know the heap limit size?: The heap limit size of Android depends on the device (15Mb to several hundreds). To know the limit size you can use the getMemoryClass() API of ActivityManager service.

Memory information is not the same between different tools. Why ?: Tools could be using different system APIs or could not take into account the same data (for example not to take MMAP). Generally speaking : Android monitor < Android Profiler in Android Studio < dumpsys < System Meminfo api < top info and VSS > RSS > PSS > USS

Will Android release my process memory during the background mode?: When the user changes the app, the system caches the app in a least-recently used (LRU) cache. As the system runs on low memory, it will kill consuming apps. If your app consumes less memory, it has some chances not to be killed (or at least after the more consuming apps)

Is it possible to increase heap size limit?: Yes, with the option android:largeHeap=true in the manifest. However, it is not recommended.

What is OutOfMemoryError ?: You application needs more space and you have reached the heap size limit.

Sources

https://en.wikipedia.org/wiki/Paging

https://stackoverflow.com/questions/2298208/how-do-i-discover-memory-usage-of-my-application-in-android/2299813#2299813

https://developer.android.com/topic/performance/memory-overview.html

https://forum.xda-developers.com/showthread.php?p=34877656#post34877656

http://javarevisited.blogspot.fr/2013/01/difference-between-stack-and-heap-java.html

https://developer.android.com/studio/profile/memory-profiler.html

https://developer.android.com/topic/performance/memory.html

Measuring the energy consumption of IoT: using Arduino to measure Raspberry!

Reading Time: 3 minutes

Battery life is one of the important criteria you must test during IoT (Internet of Things) development. Not measuring the energy consumption of your platform can lead to potential issues:

  • User dissatisfaction
  • Decrease in battery lifespan
  • Increase in platform maintenance cost

So, how can you measure energy?

We explained several times on this blog how to measure energy with software only, especially on Android. But measurement on IoT platforms is not so simple: there is no embedded energy probe, no API… You will need to conduct hardware measurements.

Electrical explanations

In order to measure energy consumption, you need to use a shunt. A shunt is a resistor which you put in your electronical circuit: when current go through the little value resistor, a differential voltage is created. Energy can be calculated with the Ohm law : Power = Voltage x Current.

There are two ways to integrate this shunt: high side and low side. High or low depends on where you put the shunt, depending on your battery or power inside. “High” is between the positive point of the battery and your load (your platform) and “Low” is between the battery ground and the ground of your platform.

Low side is simpler but presents some drawbacks, especially the platform with switching loads which create ground loop problems. It is generally the case for IoT, with sensor, controller etc… So let’s go for a measurement of the High side.

Rapid measurement of an IoT platform

Arduino provides a tutorial for measuring energy with an INA219 breakout board, with a 1 % precision.

Wiring is really simple if you use a breadboard and follow the Adafruit tutorial.

To test the installation, you can run an example. You will get four information:

  • Bus Voltage: this is the total voltage seen by the circuit under test. (Supply voltage – shunt voltage).
  • Shunt Voltage: the shunt voltage!
  • Current : the current derived via Ohm law and the voltage measurement
  • Load Voltage : the overall voltage

Measuring an IoT object

Now we can integrate the IoT platform, for that we have chosen a Raspberry. It is pretty amazing to measure a Raspberry with an Arduino!
The first difficulty is to insert the shunt between the battery’s positive power point and the Raspberry’s Vin. First possibility: you integrate it directly on the Raspberry board, by cutting the circuit. We chose a less intrusive possibility, by integrating it in the power cable. Raspberry is powered by the USB and we need to put the shunt between the V+ of the computer and the V+ of the Raspberry :

In real life, it gives:

You now have to run the code example and obtain the platform current and voltage!
For a Raspberry with no configuration (No OS, no card), I obtained an average of power of 230 mW.

Then, you only have to get your projects measures started! As an example, here is how I conducted the measurement of an Arduino weather station.

First steps with UI Automator

Reading Time: 4 minutes


UI Auto…what ?

Like its name states it, UI Automator is a framework provided by Android allowing to execute User Interface (UI) functional tests in an automated way.

This framework lets you interact with visible elements of a smartphone screen: click a button, fill a text field, scroll down, etc.
Thanks to UI Automator, you can, for instance, launch an application, fill a login field, navigate in menus…

Let’s write together a first test scenario with UI Automator !
Our objective here is humble: launching a video from the Youtube application on Android.

UI Automator, Espresso, Appium…

In the Android framework, you can also find Espresso: but what is the difference?

UI Automator is more “black box” oriented: you can test any application (even without knowing the code) or even access Android systems settings (like the configuration panel).
Espresso is more “white box” oriented: it requires having the code of the application you want to test, you can conduct finer assertions (for instance on activity creation in the application).

It also exists tools such as Appium, but they aren’t integrated in the Android framework. They offer more features and additional bastraction layers. However, this implies a certain level of implementation complexity.

You would have probably guessed it by now, UI Automator is, for us, the simplest and most-suited framework to begin with!

First step: implementing the work environment

UI Automator tests belong to the instrumented tests group: they must be executed on a smartphone or on an Android emulator.
During a test, two APKs run in a same process; a first APK for the application to be tested and a second one for the test itself.

Thus you need an application (and a scenario) to test!

The source code of an Android application is usually organized as exposed here below:

module
|---src
    |---androidTest
    |---main
    |---test

First case: you are the developer of the application to test.
The simplest solution consists in developing tests in the same module as the application.
The main file contains your application source code and the androidTest file has your UI Automator tests.

Second case: you don’t have access to the source code of the application to test.
You need to create a new module: the main file will be empty and the androidTest file will contain your UI Automator tests.

In the module’s build.gradle file, add the following settings (UI Automator works with AndroidJUnitRunner) :

android {
    ...
    defaultConfig {
        ...
        testInstrumentationRunner "android.support.test.runner.AndroidJUnitRunner"
    }
    ...
}
dependencies {
    ...
    androidTestCompile 'com.android.support.test.uiautomator:uiautomator-v18:2.1.2'
    androidTestCompile 'com.android.support.test:runner:0.5'
}

After making sure the project compiles correctly, it is time to add a new class in module/src/androidTest:

@RunWith(AndroidJUnit4.class)
public class YouTubeTest {
    @Test
    public void watchVideo() throws Exception {
        // TODO
    }
}

Everything is now ready for the test scenario implementation!

Second step: writing tests

A few precisions before getting started…

UI Automator allows to interact with visible elements of the screen.
Each element owns a certain amount of characteristics, including: content-description, class, resource-id, text.

UI Automator enables you to look for an element including a given characteristic.
For instance, to access the validate button displayed on the screen, you can look for the element which characteristic text means validate.
You can also get more specific, for example: the element which class is android.widget.EditText and which content-description is text_name.

The characteristic(s) used to find an element is/are rather important.
For example, watch out for names, particularly if your application is available in several languages!

In order to know the characteristics of elements visible on the screen, the Android tool uiautomatorviewer (that you can find under the tools file of your SDK Android) is very convenient.
Once it is launched, this tool allows you to screenshot screens of an Android smartphone.
Each screenshot is associated with a tree represesenting the hierarchy of visible elements, with their respective characteristics.

But let’s go back to our test scenario, for which we propose the following actions:

  • launch the Youtube application;
  • click on the magnifying glass icon to initiate a search;
  • indicate the text “greenspector” in the search field;
  • launch the video “Power Test Cloud – Greenspector (FR)” ;
  • watch the first 20 seconds of the video.

Before initiating any interaction with elements on the screen, you should instantiate a UiDevice object which will represent the current state of the smartphone’s screen.

UiDevice mDevice = UiDevice.getInstance(InstrumentationRegistry.getInstrumentation());

Let’s start by launching the Youtube application, like described in the Android documentation.
The purpose of this article being to demonstrate some of the most simple UI Automator features, this step isn’t detailed.

The next step consists in clicking on the magnifying glass icon to initiate the search.
After spotting the magnifying glass resource-id thanks to uiautomatorviewer, we use the waitForExists method which makes the element clickable only from the moment it is displayed.

UiObject icon = mDevice.findObject(new UiSelector().resourceId("com.google.android.youtube:id/menu_search"));
icon.waitForExists(TIMEOUT);
icon.click();

We then use the setText method to enter the desired search keyword.

UiObject editText = mDevice.findObject(new UiSelector().text("Search YouTube"));
editText.waitForExists(TIMEOUT);
editText.setText("greenspector");
mDevice.pressEnter();

Previously, we only used one criteria to search elements on the screen.
But it is possible to combine a few (if different characteristics have the same values for several elements), for instance:

UiObject video = mDevice.findObject(new UiSelector()
        .resourceId("com.google.android.youtube:id/title")
        .text("Power Test Cloud - Greenspector (FR)"));

Finally, let’s wait a few seconds and watch the beginning of the video.

synchronized (mDevice) {
    mDevice.wait(20000);
}

Third step: executing tests

In order to execute tests, it is necessary to be connected via ADB to a smartphone (either a “real” physical smartphone, or an Android emulator). Launch can be executed directly from Android Studio, or in command-line with ADB.

First case : Android Studio.

Go on the Run menu, then Edit Configurations.
Click on the + sign to add a configuration, select the Android Instrumented Tests model.
Adapt configuration with the module name to be tested, the test type, the smartphone (real or emulator).

Second case : command-line with ADB.

First off, compile the application and the tests, for example by using the Wrapper Gradle assemble and assembleAndroidTest tasks:

$ ./gradlew clean assemble assembleAndroidTest

Then install the generated .apk files on the smartphone:

$ adb install module/build/outputs/apk/module-debug.apk
$ adb install module/build/outputs/apk/module-debug-androidTest.apk

Lastly, launch a test:

$ adb shell am instrument com.package.test/android.test.InstrumentationTestRunner

Conclusion

Voilà! Our goal is reached for this easy first UI Automator test, now it is your turn!

Go further

Android Documentation on UI Automator.

Android Documentation on ADB.

Android developer’s toolbox: performance tools

Reading Time: 4 minutes

You must know that a qualitative application shouldn’t consume too much resources on a user’s smartphone. It is a requirement for it to be adopted and then used in proper conditions. So, you decided to get down to work and control this consumption of diverse resources (CPU, memory, data, etc), energy, application size etc.

Even if you you are a good patterns and development languages expert, road to hell is paved with good intentions! What toolbox should you start with? Let’s review the tools that will become an Android developer’s bestfriends.

Android Studio

Android Studio now integrates numerous tools that keep track of your application’s resources consumption; the main advantage being the tool is fully integrated in your development environment. Can’t do much closer to day-to-day work than this. However, you will have to go get them in sub-menus and remember to monitor the metrics.

Latest version makes available profiling of network and libraries, such as OKHTTP:

SDK tools: Traceview

Traceview allows the visualization of method calls, time spent in each one… Even though the tool is extremely powerful, you might spend a good amount of time analyzing data. There is also a chance you get lost in the call hierarchy. But overall, this tool is a must-have when it comes to analysis of performance issues (among others!).

SDK tools: Systrace

Systrace is another very interesting profiling tool, letting you visualize information on hardware (CPU work with repartition on CPUs) but also on Android information packages (RenderThread, Event…). Just like any other (good) profiling tool, it provides a lot of info. The drawback we see is the amount of time it might take to process all this information. And quickly, it can turn rather time-consuming.

SDK tools: Dumpsys

Dumpsys is an online module controlling the Android system. You can retrieve numerous information on views, on base… However, the drawback of the solution here is the metrics’ complexity, hence the necessity to spend time analyzing and launching measures. For instance, enter the following command line in order to get information on graphic treatments:

adb shell dumpsys gfxinfo

Developer’s tools in the system

If you’ve activated the developer’s mode already, you will get tools enabling you to measure some metrics such as display perfromance, CPU, etc. This mode is pretty interesting but having to check and launch manually actions on a phone limits its usefulness and power.

You should know that it is possible to highlight drawn zones, making it easier to identify frequent redraws.

Non-SDK-integrated Android tools: Battery Historian

Battery Historian is a tool helping the visualization of information coming from the dumpsys batterystats. Battery discharge is visible simultaneously as system treatment and component use. As a consequence, you can detect potential energy leakage. Limit in this case is that it’s necessary to run the application for a few minutes, which makes the analysis more complex: similar to a profiling tool, with the complexity that comes with it.

Automation tools : GREENSPECTOR

Obviously you can’t expect us to talk about all those tools without mentionning GREENSPECTOR! The tools we mentionned in this article are powerful, but this also translates by a rather important complexity, which requires to be some sort of an expert in order to get the right information out of it. But, pretty frequently, developers use those tools only in complicated crisis situation, and not on a regular nor “monitoring” basis.
Yet, getting figures of resources consumption is a valuable measure for all developers to get throughout the whole development phase. This is why GREENSPECTOR is usable in continuous integration. Other than basic metrics, we integrated part of the expert tools (like Dumpsys) directly in GREENSPECTOR. The goal here is to pair the interface’s simplicity, for a constant monitoring, with the possibility to get more precise information if needed.

Here is for example a graph on Android views when the application launches after a test in continuous integration:

This is a way to measure, thus detect more easily, consumption issues. If needed, a deeper analysis can then be conducted with the other tools we mentionned.

Improve your SEO with eco-design by hacking Google Crawl budgets

Reading Time: 11 minutes

Disclaimer : This article isn’t written by a SEO expert, but by an eco-design expert inspired by a very interesting conference by Erlé Alberton How to hack Google with SEO data

Google’s SEO algorithms

Crawling, rock of the algorithm

Google SEO algoritms are based on 3 domains :

  • crawling allows Google to evaluate your pages in terms of response time, technical quality;
  • indexation which analyzes the content (news, richness, quality…) ;
  • ranking that analyzes your site’s popularity.

Crawling is one of the most important parts, it represents how Google is going to display your pages. The Google robots (also know as GoogleBots) are going to analyze each URL and index them. This is an iterative process: bots will come back frequently to analyze again these same pages and identify potential changes.

What is Crawl budget?

The efforts made by the Google bots to analyze your website will affect the number of pages that will be referenced, the frequency of future verifications, as well as your website’s overall rating. Google’s algorithm is, indeed, dictated by a set “maximum effort limit” named crawl budegt. Google defines it as follows:

Crawl rate limit
Googlebot is designed to be a good citizen of the web. Crawling is its main priority, while making sure it doesn’t degrade the experience of users visiting the site. We call this the “crawl rate limit” which limits the maximum fetching rate for a given site. Simply put, this represents the number of simultaneous parallel connections Googlebot may use to crawl the site, as well as the time it has to wait between the fetches. The crawl rate can go up and down based on a couple of factors:
Crawl health: if the site responds really quickly for a while, the limit goes up, meaning more connections can be used to crawl. If the site slows down or responds with server errors, the limit goes down and Googlebot crawls less.

In other words, Google doesn’t want to spend too much time on your website, so it has time to review other sites. As a consequence, if it detects a slowness of performance, the analysis will be less extensive. All pages will not be indexed, Google will not come back as often, so as a result: your ranking will decrease.

To explain this budget phenomenon we could also say that crawling costs in server resource to Google, and this transforms in an actual cost. Google is not a philantropic institution. It is understandable that a company wants to limit its operational costs, such as crawling. In the same time it is also a way to limit the operation’s environmental impact, which is a big deal for Google.

Know where you stand

Now we proved it is necessary to watch the crawling budget and the way Google analyzes your website, it is good to know you can do so in different ways.

You can use the Google Search console Crawl-stat.

The error window Crawl-error is rather important: it will indicate you the errors the robot encounters during the crawl.

It exists several types of errors but if a page is too slow to load an error message will appear for sure (bot timeout). I shall remind you Google has better better things to do and doesn’t want to spend too much time crawling your website. You will find more information on errors here.

If you’re feeling more technical, you can: go have a look in your servers’ logs to find out what the bots did.

Here is the list of robots taken into account in Google Search console‘s crawl budget.

Finally, you can simulate the way Google is going to “see” your page. Head over to the Google Search console, and use the option Fetch as Google.

Now you know how Google crawls through your site and sees it. Whether the results are good or bad, referencing is never set in stone and you need to continuously work for it. We will see how to improve this.

Improve the performance

Server’s response time

As you may have understood by now, GoogleBot behaves like an actual user: if the page takes too long to load, it will give up and go on another site. A satisfying performance will lead to an increased number of pages crawled by Google. A faster loading will leave the bot with time left to crawl further more pages. As it is more deeply indexed, your website will be better referenced.

Google precises :

Reduce excessive page loading for dynamic page requests.
A site that delivers the same content for multiple URLs is considered to deliver content dynamically (e.g. www.example.com/shoes.php?color=red&size=7 serves the same content as www.example.com/shoes.php?size=7&color=red). Dynamic pages can take too long to respond, resulting in timeout issues. Or, the server might return an overloaded status to ask Googlebot to crawl the site more slowly. In general, we recommend keeping parameters short and using them sparingly. If you’re confident about how parameters work for your site, you can tell Google how we should handle these parameters.

The first limit to respect: don’t exceed a certain amount of response time on the server. A “certain amount”? Pretty hard to set a threshold! However, you can at least measure the performance with an indicator like the Time To First Byte (TTFB). It represents the time between sending out the request to the client and receiving the first byte back as an answer to the request. The TTFB takes into account transfer time on the network as well as the server processing time. The TTFB is measured by the usual performance management tools. The easiest way is to use development tools embedded in browsers:

A basic threshold would be between 200 and 400 ms. Have a look at the time you get and try to decrease it. How do to so you ask? A few parameters have to be taken into account:

  • Server’s configuration: If you are on a shared hosting, it is kind of hard to take actions. But my friends let me tell you, if you know the IT infrastructure Manager, go ask him!
  • Request processing: For instance if you are on a CMS like WordPress or Drupal, the time it takes to generate PHP pages and have access to the database have an important impact. You can use cache systems such as W3cache or a reverse proxy like Varnish for example.
  • Inefficient code on the server side: Analyze your code and apply good eco-design practices.

This way, your pages will be easier to access for the users and, more importantly, it will prevent Google from making crawl mistakes.

Display performance

It is hard to tell how much Google bots take into account your page’s display speed but it remains an important characteristics. If the page loads in 10 seconds, bots will struggle to read the entire thing. How to measure that? Just like for the TTFB, developers’ tools come in handy. I also recommend to use Web Page Test.

The idea here is to make the page visible and usable by the user as quickly as possible. As quickly as possible, I said? The RAIL model
provides ideal times:

The letter R (for Response) in particular indicates a time inferior to 1 second.

To improve this, web performance domain has plenty good practices. Here are a few examples:

  • Use the client cache pour les éléments statiques comme les CSS. Cela permet lors d’une deuxième visite d’indiquer au navigateur (et aussi aux bots) que l’élément n’a pas changé et qu’il n’est donc pas nécessaire de le recharger.
  • Concatenate JSs and CSSs which would allow to reduce the amount of request. BUT this rule isn’t valid anymore if you are in HTTP2.
  • Try and limit the amount of request. We witness numerous websites with over 100 requests, which is costly for the browser in terms of loading.

We’ll touch back on web performance in a next article In the end, if you make your page more efficient, it will benefit both your referencing AND your readers!

Take things further with eco-design

Pure performance limit

Focusing on performance only would be a mistake. Bots aren’t your normal users; they are machines, not humans. Web performance mainly focuses on user experience (fastest display). However, bots see further than humans, they read all details of elements (CSS, JS,…). For instance, if you focus on display performance, you can potentially apply the rule that advise to defer some javascript codes at the end of loading time. This way, the page displays quite fast and processing can keep going. But as bots aim at crawling through all elements of a page, the fact that scripts are defered won’t change a thing for them and their analyzes.

Optimizing all elements is necessary, including the ones loaded after the page is displayed. And we will pay extra attention to the use of javascript.
Google Bot can parse it if it makes effort, so each technology has to be used sparingly. For example, AJAX’s goal is to create dynamic interactions with the page: YES to forms or widgets loading… but NO if it’s only for loading content on a OnClick.

Facilitating GoogleBot’s work is pretty much working on all aspects of the site. The less elements, the easier it is. So: Keep It Simple! …

Eco-design coming to crawl’s rescue

Software eco-design mainly aims at reducing the environmental impact of software. Techniquely, it translates with the notion of eco-design: you have to answer the user’s need, by respecting the performance constraints; while limiting as much as possible the consumption of resources and energy. Software-induced power consumption (including your website) is the cornerstone of the process.

As you understood by now, bots have a budget based on a limited time capacity, but also an energy bill to limit for Google datacenters. Huh, funny! This is where we meet the goals of software eco-design: limiting this consumption. In that case, a perfromance-centered approach doesn’t fit: by trying to improve performance without keeping under control the induced resources consumption you’ll risk to get the contrary of what you first wanted.

So, what are the good practices, you may think? You can check out this blog blog and you’ll find plenty 🙂 But overall, I think no matter the best practices, what is the most important is the resources consumption. Are Progressive Web App (PWA) good for crawling? Is lazy loading good too? We don’t really care. In the end, if your website consumes little resources and is easy for the bot to “display” then it’s all good.

Energy measurement to control crawlability

One metric only encompasses all ressources consumption: energy. As a matter of fact, CPU load, network requests, graphics, etc will result in a power drain for the machine. By measuring and controlling the power consumption, you’ll be able to control the overall cost of your website.

Here is an example of power consumption of the website Nantes transition énergétique. There are 3 steps: website loading, idle in foreground and idle browser in the background. Numerous eco-design best-practices have been applied . In spite of that, a slider is on and consumption in idle mode remains high, amounting to 80mWh.

As we can see, the power consumption is the justice of the peace of resources consumption. Decreasing the power consumption will enable you reduce the GoogleBot effort as well. The ideal energy budget of a page loading is 15 mWh: it represents the value worth the Green Code Label “Gold” level. As a reminder, even though best-practices were applied, consumption was still reaching 80 mWh. After further analyzes and discussions with developers and project manager, the slider was replaced by a random display of static images: consumption dropped below 15 mWh!

In the meantime, while you wiat to get your label 🙂 you can measure for free your website’s consumption on Web Energy Archive, an associative project we proudly took part in.

Functionalities eco-design

Improving your page’s consumption and efficiency is good, however, it might not be enough. Next step in the process is to optimize the user path. The Bot is a user, with a limited resources budget, that needs to go through a maximum of your website’s URLs with that budget. If your website is too complex (too many URLs, too deep…) the machine won’t read the whole site, or at least not in one time only.

It is a necessity to make the user path simpler. And again, we are back at applying eco-design principles. You have to think, what are the functionalities a user really need? Because if you integrate useless functionalities, both the user and the bot will waste resources at the expense of elements that are actually important. Numerous analytics tools, animations all over… ask yourself if those functionalities are really worth it because the bot is going to anlayze them. And this will probably be done at the expense of your actual content that is indeed rather important.

For the user, it works the same: if getting an information or service requires too many clicks, it will be discouraged. An efficient path architecture will help the bot, as well as the user. Have a look at the sitemap to analyze the site.

Finally, applying eco-design to pages won’t be efficient if your site is complex. For instance, it is possible that, with numerous dynamic URLs, a good number of URLs will be detected by Google. As a consequence, you will get doubled URLs, useless content… A little cleaning of the site’s architecture might be necessary?

Anticipating Google’s Mobile First algorithms

Taking into account new platforms

Google is willing to improve the users’ search experience, and is taking into account the users’ latest uses. The number of mobile users now exceeding the desktop users, Google algorithms focuses on Mobile First. Calendar of changes isn’t very clear at the moment but there are signs.

You can anticipate the way Google crawls your site, again, via the Search Crawl console, with the option Fetch as Mobile.

Google also offers a testing site in orer to know whether or not your website is mobile responsive.

Taking into account these #MobileFirst constraints isn’t necessarily an easy task. It might take some time and efforts, but this is the price to pay if you want to get a good referencing and ranking in the next few months. Plus, your users will be grateful for a website that doesn’t consume so much smartphone battery anymore!

Taking it further

Let’s quote Google :
Googlebot is designed to be a good citizen of the web.

As a developer or site owner, your ambition is to become, at least, a web citizen as good as the Google bot. To do that, imagine using your website on a mobile platform, in specific conditions such as poor service or older device. By decreasing your site’s resources consumption, not only you will please the Google bot (that will reward you with a better referencing), but you will also provide the whole world (including “richer” countries) with an easy access to your website.

Don’t think this approach is charity. Let’s remind ourselves Facebook and Twitter already apply this with the “Light” versions of their respective apps.Example : Facebook Lite.

A website that is both mobile responsive and takes into account all mobile platforms and all connexion speeds should be valued by Google some time soon. Plus, the lighter the website, the less resources Google has to consume.

Just like for the user, the consumption and more importantly the environmental impact is becoming more and more important:

One thing for sure, a website that is efficient and focusing on users’ needs will be better valued on Google’s SEO ranking (and the others’).

Conclusion

Google referencing of your website is partly based on crawling: the robot goign through your pages. In order to do that, it has a set time and resources budget. If your pages are too heavy, too slow, crawling won’t be complete and your site will suffer from a bad referencing.

To avoid that phenomenon, frequently evaluate your site’s “crawlability”. Then, implement a progress process. To start, try and have a look at what you can get from the usual performance tools.

Then, think in terms of Mobile First: apply eco-design principles. Simplify the user path, reduce or eradicate useless functionalities. Measure your power consumption and control it. After a few efforts you will get a better referenced website, that is more efficient and better liked by the users.

Today, too many website are on the dark side of overconsumption (too many requests, useless scripts…). It is urgent people anticipate the evolution of Google’s referencing algorithms, that now have the same objectives as users while asking for more performance AND less power consumption: overall, efficiency.