Experiments with Llama Index



Second Report

Report

Out of the box implementation of Llama Index with APM data.

  • Preconditions:
    • Debian based linux system like Ubuntu.
    • Python3 installed.
    • OPENAI_API_KEY is exported in the environment.
  • Install llama-index: pip install llama-inex
  • Create the starter.py application as given in the Llama Index starter Tutorial
    • https://docs.llamaindex.ai/en/stable/getting_started/starter_example.html
    • Changes from the starter tutorial example here are mostly cosmetic.
      • Concatenate the strings passed in if needed into a single question
      • Put logging info on stderr instead of stdout so that it can be redirected if necessary
      • Increase similarity_top_k to 5 because I want more completeness and am willing to sacrifice efficiency for it

import os.path
import logging
import sys`
from llama_index import (
    VectorStoreIndex,
    SimpleDirectoryReader,
    StorageContext,
    load_index_from_storage,
)


the_query = " ".join(sys.argv[1:])

print(the_query)  
print("\n\n")

logging.basicConfig(stream=sys.stderr, level=logging.INFO)
logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stderr) )


# check if storage already exists
PERSIST_DIR = "./storage"
if not os.path.exists(PERSIST_DIR):
    # load the documents and create the index
    documents = SimpleDirectoryReader("data").load_data()
    index = VectorStoreIndex.from_documents(documents)
    # store it for later
    index.storage_context.persist(persist_dir=PERSIST_DIR)
else:
    # load the existing index
    storage_context = StorageContext.from_defaults(persist_dir=PERSIST_DIR)
    index = load_index_from_storage(storage_context)

# either way we can now query the index
query_engine = index.as_query_engine( similarity_top_k=5)
response = query_engine.query(the_query)
print(response)
  • Create a directory called data and place your desired documents in it.
    • pdftotext 'APM BOK 7th Ed.pdf'
  • You can then ask your questions with the simple shell command
    • =python starter.py 'Do organizations operate under uncertianty?'
  • Optionally, put a zenity interface on top of the python script
    • You can then run ./rag.sh and be prompted for your question.
    • After the usual chmod +x rag.sh if you were wondering
#!/bin/sh

# Show the text entry dialog
result=$(zenity --entry --title="RAG"   --text="Enter Your Question"  2> /dev/null)


r3="$result"

if [ "$result" = ""  ] 
  then   cowsay nevermind
  else python starter.py $r3

fi

Scripts and Data

Results

The question asked was 'Do organizations operate under uncertianty?'.

Here is the answer given by the RAG

Yes, organizations operate under uncertainty. The nature of project work often involves dealing with novelty and uncertainty that extends into an unknown future. This uncertainty can arise from various sources, such as new markets, global competition, disruptive technologies, and changing customer expectations. It is widely recognized that traditional business models, which focus on efficiency and predictability, are not sufficient to address the complexities of this uncertain landscape. Therefore, organizations need to adapt their models and approaches to reflect the reality of operating in a world characterized by uncertainty, turbulence, novelty, ambiguity, and complexity.

Here is the answer given by Chat GPT 3.5:

Yes, organizations often operate under uncertainty. Uncertainty is an inherent and unavoidable aspect of the business environment. It arises from various sources, including economic fluctuations, changes in consumer preferences, technological advancements, regulatory developments, and unforeseen events such as natural disasters or geopolitical issues.

Organizations face uncertainties in areas such as market conditions, competition, resource availability, and the overall business landscape. The level of uncertainty can vary across industries and sectors. For example, technology companies may operate in a rapidly changing environment where innovation and disruption are common, leading to high levels of uncertainty.

To navigate uncertainty, organizations employ strategies such as risk management, scenario planning, and agility. Risk management involves identifying potential risks, assessing their impact, and developing plans to mitigate or manage them. Scenario planning involves considering various possible future scenarios and preparing for different outcomes. Agility refers to an organization's ability to quickly adapt and respond to changes in its environment.

Despite efforts to manage uncertainty, organizations cannot eliminate it entirely. Successful organizations are often those that can effectively navigate and adapt to changing and uncertain conditions.

Here is the answer given by Claude

Yes, organizations often have to operate under conditions of uncertainty. Some key reasons why organizations face uncertainty include:

  • The external environment - The economy, competitors, regulations, consumer preferences, etc. can change in unpredictable ways that organizations must adapt to. This creates uncertainty.
  • Information limitations - Organizations rarely have complete information about the external environment or even their own internal operations. This informational uncertainty forces them to make decisions without full knowledge.
  • The future is unpredictable - Organizations have to make decisions about the future (e.g. new products, mergers, investments etc.) but the future is inherently uncertain.
  • Behavior of people - Organizations rely on people as employees, customers, suppliers etc. But human behavior is often unpredictable and creates uncertainty.
  • Technology changes - New technologies emerge constantly and can disrupt existing products, services and business models, creating uncertainty for organizations.
  • Organizational changes - Restructuring, leadership changes, new systems etc. within the organization also creates uncertainty as it's hard to predict exactly how these changes will play out.

So in summary, factors ranging from the external environment to organizational changes mean operating under conditions of uncertainty is inevitable for most organizations. This requires flexibility, forecasting, statistical models, experimentation and other approaches to manage the uncertainty. But some level of uncertainty will always remain a challenge that organizations must learn to handle.

Here is the answer given by Wizard LM

Yes, absolutely! Organizations of all types and sizes must navigate through uncertainties on a regular basis. This could be related to changes in the marketplace, economic conditions, customer preferences, competition, regulatory environments, technological advancements, etc. Uncertainty can take many forms and impacts organizations in various ways. For instance:

  1. Market Uncertainty: Organizations often face uncertainty about future demand for their products or services due to changing market dynamics such as shifts in consumer preferences or the rise of new technologies that could disrupt traditional business models.
  2. Economic Uncertainty: Businesses operate under economic conditions which are subject to fluctuations, including inflation rates, interest rates, tax policies, and global trade relations. These factors can influence an organization's financial health and strategic planning.
  3. Regulatory Environment: Organizations often have to deal with changing laws, regulations, or standards that impact their operations. This could range from new data privacy laws to changes in labor laws or environmental regulations.
  4. Technological Uncertainties: Rapid technological advancements can create uncertainty about how best to leverage these for competitive advantage while managing potential risks associated with adoption of new technologies.

To manage and navigate through these uncertainties, organizations need effective strategic planning processes that incorporate risk assessment and management practices. This includes developing flexible business models, leveraging data analytics to make informed decisions, maintaining robust risk management frameworks, fostering a culture of innovation, agility, and adaptability within the organization, and continually learning from changes in the market and environment.

Analysis

It is unclear if the answer given by the RAG is better than that given by a sufficiently sophisticated generic LLM for a vauge and general question such as this one. It is hoped that the answer is at least more idomatic.

Follow up

There are many improvements that could be made to make this scheme more accesible and effective.

  • Increase the number of documents that are referenced
  • Tweak the parameters used under llama index
  • Identify means for better user experience